2017-10-01 00:00:36,413-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [2c0115bb] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 00:01:36,430-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [7307f9c4] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 00:02:36,454-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [9137134] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 00:03:36,473-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [12708435] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 00:04:36,493-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [7c9b5759] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 00:05:36,513-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [14659c0d] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 00:06:36,532-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [266eda04] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 00:07:36,557-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler7) [1f8addd3] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 00:08:36,580-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [5094129b] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 00:09:36,597-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [7dae3f51] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 00:10:36,615-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [7e45ae3c] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 00:11:36,635-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [415420d8] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 00:12:36,659-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [7032d320] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 00:13:36,678-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [31d3fca7] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 00:14:36,697-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [3df713ba] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 00:15:36,715-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [385b8871] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 00:16:36,735-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [477719eb] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 00:17:36,759-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [1907b35d] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 00:18:36,778-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [1218dd2f] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 00:19:20,451-04 INFO [org.ovirt.engine.core.bll.EngineBackupAwarenessManager] (DefaultQuartzScheduler10) [7f20f294] Backup check started. 2017-10-01 00:19:20,454-04 INFO [org.ovirt.engine.core.bll.EngineBackupAwarenessManager] (DefaultQuartzScheduler10) [7f20f294] Backup check completed. 2017-10-01 00:19:36,798-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [75b4ee52] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 00:20:24,169-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.OvfDataUpdater] (DefaultQuartzScheduler5) [40e5d60e] Attempting to update VMs/Templates Ovf. 2017-10-01 00:20:24,171-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler5) [3ee73525] Before acquiring and wait lock 'EngineLock:{exclusiveLocks='[aca61eac-ed8a-484b-bbee-3645148f6405=OVF_UPDATE]', sharedLocks=''}' 2017-10-01 00:20:24,172-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler5) [3ee73525] Lock-wait acquired to object 'EngineLock:{exclusiveLocks='[aca61eac-ed8a-484b-bbee-3645148f6405=OVF_UPDATE]', sharedLocks=''}' 2017-10-01 00:20:24,173-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler5) [3ee73525] Running command: ProcessOvfUpdateForStoragePoolCommand internal: true. Entities affected : ID: aca61eac-ed8a-484b-bbee-3645148f6405 Type: StoragePool 2017-10-01 00:20:24,178-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler5) [3ee73525] Attempting to update VM OVFs in Data Center 'EECS-Primary' 2017-10-01 00:20:24,178-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler5) [3ee73525] Successfully updated VM OVFs in Data Center 'EECS-Primary' 2017-10-01 00:20:24,178-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler5) [3ee73525] Attempting to update template OVFs in Data Center 'EECS-Primary' 2017-10-01 00:20:24,179-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler5) [3ee73525] Successfully updated templates OVFs in Data Center 'EECS-Primary' 2017-10-01 00:20:24,179-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler5) [3ee73525] Attempting to remove unneeded template/vm OVFs in Data Center 'EECS-Primary' 2017-10-01 00:20:24,180-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler5) [3ee73525] Successfully removed unneeded template/vm OVFs in Data Center 'EECS-Primary' 2017-10-01 00:20:24,181-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler5) [3ee73525] Lock freed to object 'EngineLock:{exclusiveLocks='[aca61eac-ed8a-484b-bbee-3645148f6405=OVF_UPDATE]', sharedLocks=''}' 2017-10-01 00:20:36,818-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [31e70d97] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 00:21:36,838-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [41716a0f] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 00:22:36,864-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler7) [31bf3989] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 00:23:36,886-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [1b71cd85] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 00:24:36,906-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [4aaa3895] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 00:25:36,926-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [77bede2b] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 00:26:36,945-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [6887f4c3] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 00:27:36,969-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [4720c022] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 00:28:15,629-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-6-thread-10) [51799762] START, GetFileStatsVDSCommand( GetFileStatsParameters:{runAsync='true', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', ignoreFailoverLimit='false'}), log id: 26433ba1 2017-10-01 00:28:16,642-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-6-thread-10) [51799762] FINISH, GetFileStatsVDSCommand, return: {virtio-win-0.1.126.iso={status=0, ctime=1497024605.0, size=155856896}, server2012r2_virtio.iso={status=0, ctime=1497297085.0, size=4977590272}, en_windows_7_professional_with_sp1_x64_dvd_621750.iso={status=0, ctime=1435288543.0, size=3319478272}, virtio-win-0.1.105.iso={status=0, ctime=1435288648.0, size=56584192}}, log id: 26433ba1 2017-10-01 00:28:16,646-04 INFO [org.ovirt.engine.core.bll.storage.domain.IsoDomainListSynchronizer] (org.ovirt.thread.pool-6-thread-10) [51799762] Finished automatic refresh process for 'ISO' file type with success, for storage domain id '63d680c7-4c94-4b29-a6cb-f288a128b9bc'. 2017-10-01 00:28:36,990-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [208de395] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 00:29:37,009-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [6df499aa] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 00:30:37,028-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [472d60dd] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 00:31:37,048-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [27747b13] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 00:32:37,073-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [6baf9ae2] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 00:33:37,092-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [5bba849b] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 00:34:37,112-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [786f4700] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 00:35:37,131-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [4ed682af] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 00:36:37,150-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler7) [ae0bcfb] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 00:37:37,175-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [65fb878] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 00:38:37,194-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [1b83fd16] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 00:39:37,216-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [72af0e67] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 00:40:37,233-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [339f0064] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 00:41:37,251-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [4a77c08b] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 00:42:37,275-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [2b187fc0] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 00:43:37,294-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [2746305] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 00:44:37,314-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler7) [228fe657] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 00:45:37,334-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [68adb48e] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 00:46:37,351-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [46ec65dc] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 00:47:37,376-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [59353446] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 00:48:37,395-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [f922e87] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 00:49:37,415-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [680eab02] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 00:50:37,435-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler7) [2ee9a064] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 00:51:37,456-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [291cdd18] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 00:52:37,481-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [6f79bf68] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 00:53:37,503-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [361cd373] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 00:54:37,523-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler7) [6f2c59e1] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 00:55:37,546-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [28b6d916] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 00:56:37,565-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler7) [5f3bcf5b] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 00:57:37,589-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [49cb52ad] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 00:58:37,610-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [7db31a38] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 00:59:37,632-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [47f2c610] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 01:00:37,653-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [785fd86d] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 01:01:37,673-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [254f339d] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 01:02:37,697-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [56664f30] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 01:03:37,714-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [3885d5fb] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 01:04:37,735-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [4aad4647] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 01:05:37,754-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [30d3efcc] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 01:06:37,774-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [6561711d] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 01:07:37,797-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [27de1cb9] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 01:08:37,817-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [6ed5eb76] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 01:09:37,838-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [2837e79] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 01:10:37,861-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [33c98c13] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 01:11:37,880-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [1c85e554] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 01:12:37,903-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [596fc981] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 01:13:37,922-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [2c55c14c] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 01:14:37,942-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [57e88fa2] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 01:15:37,964-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [2695c95c] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 01:16:37,982-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [47afb764] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 01:17:38,006-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [7390229f] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 01:18:38,025-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler7) [3b21b887] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 01:19:38,045-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [2b258979] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 01:20:24,182-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.OvfDataUpdater] (DefaultQuartzScheduler6) [c5509c5] Attempting to update VMs/Templates Ovf. 2017-10-01 01:20:24,184-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [6b6207a7] Before acquiring and wait lock 'EngineLock:{exclusiveLocks='[aca61eac-ed8a-484b-bbee-3645148f6405=OVF_UPDATE]', sharedLocks=''}' 2017-10-01 01:20:24,184-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [6b6207a7] Lock-wait acquired to object 'EngineLock:{exclusiveLocks='[aca61eac-ed8a-484b-bbee-3645148f6405=OVF_UPDATE]', sharedLocks=''}' 2017-10-01 01:20:24,186-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [6b6207a7] Running command: ProcessOvfUpdateForStoragePoolCommand internal: true. Entities affected : ID: aca61eac-ed8a-484b-bbee-3645148f6405 Type: StoragePool 2017-10-01 01:20:24,191-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [6b6207a7] Attempting to update VM OVFs in Data Center 'EECS-Primary' 2017-10-01 01:20:24,191-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [6b6207a7] Successfully updated VM OVFs in Data Center 'EECS-Primary' 2017-10-01 01:20:24,191-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [6b6207a7] Attempting to update template OVFs in Data Center 'EECS-Primary' 2017-10-01 01:20:24,192-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [6b6207a7] Successfully updated templates OVFs in Data Center 'EECS-Primary' 2017-10-01 01:20:24,192-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [6b6207a7] Attempting to remove unneeded template/vm OVFs in Data Center 'EECS-Primary' 2017-10-01 01:20:24,193-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [6b6207a7] Successfully removed unneeded template/vm OVFs in Data Center 'EECS-Primary' 2017-10-01 01:20:24,194-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [6b6207a7] Lock freed to object 'EngineLock:{exclusiveLocks='[aca61eac-ed8a-484b-bbee-3645148f6405=OVF_UPDATE]', sharedLocks=''}' 2017-10-01 01:20:38,064-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [77e62f69] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 01:21:38,083-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [27f8114e] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 01:22:38,108-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [6f1d6173] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 01:23:38,127-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [57aa7d7b] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 01:24:38,144-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [347da1eb] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 01:25:38,163-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [5b10013d] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 01:26:38,182-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [6d253211] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 01:27:38,208-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [1f47c8dc] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 01:28:16,649-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-6-thread-21) [49c55f3] START, GetFileStatsVDSCommand( GetFileStatsParameters:{runAsync='true', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', ignoreFailoverLimit='false'}), log id: 21530884 2017-10-01 01:28:17,500-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-6-thread-21) [49c55f3] FINISH, GetFileStatsVDSCommand, return: {virtio-win-0.1.126.iso={status=0, ctime=1497024605.0, size=155856896}, server2012r2_virtio.iso={status=0, ctime=1497297085.0, size=4977590272}, en_windows_7_professional_with_sp1_x64_dvd_621750.iso={status=0, ctime=1435288543.0, size=3319478272}, virtio-win-0.1.105.iso={status=0, ctime=1435288648.0, size=56584192}}, log id: 21530884 2017-10-01 01:28:17,503-04 INFO [org.ovirt.engine.core.bll.storage.domain.IsoDomainListSynchronizer] (org.ovirt.thread.pool-6-thread-21) [49c55f3] Finished automatic refresh process for 'ISO' file type with success, for storage domain id '63d680c7-4c94-4b29-a6cb-f288a128b9bc'. 2017-10-01 01:28:38,227-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [4c85ee76] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 01:29:38,244-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [3074d1d3] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 01:30:38,263-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [3e35bead] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 01:31:38,285-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [6eddb34c] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 01:32:38,309-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [7d1e0ae1] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 01:33:38,328-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [6e8c6abf] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 01:34:38,348-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [2a7ca5f8] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 01:35:38,367-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [34dcecfd] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 01:36:38,386-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [4062d9e5] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 01:37:38,410-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [1de169ba] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 01:38:38,429-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [694b3564] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 01:39:38,448-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [2be27765] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 01:40:38,468-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [326261ae] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 01:41:38,490-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [24e105fc] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 01:42:38,512-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [74c1a98f] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 01:43:38,531-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler7) [1f325804] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 01:44:38,550-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [3453436f] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 01:45:38,569-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler7) [136b87c5] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 01:46:38,588-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [d7a6049] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 01:47:38,612-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [119193fb] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 01:48:38,632-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [74b90faf] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 01:49:38,651-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler7) [27747ed0] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 01:50:38,673-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [2422fbc6] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 01:51:38,692-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [58e24fd2] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 01:52:38,717-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [1119a061] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 01:53:38,736-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [27c383ed] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 01:54:38,755-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [cbeea4e] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 01:55:38,774-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [2d62ef4a] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 01:56:38,794-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [54bf8797] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 01:57:38,818-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler7) [4d7f8911] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 01:58:38,838-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler7) [599adfeb] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 01:59:38,857-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [756c44f0] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 02:00:38,877-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [5d275614] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 02:01:38,895-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [70a9fd9a] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 02:02:38,919-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [46fbc1ed] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 02:03:38,937-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [1bfc74e4] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 02:04:38,956-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [3d43f43] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 02:05:38,975-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [7c92902e] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 02:06:38,995-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [4a8aea51] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 02:07:39,016-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [339098be] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 02:08:39,035-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [4be184b6] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 02:09:39,052-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [51caa9dd] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 02:10:39,075-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [4df317b9] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 02:11:39,094-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [430dc5fe] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 02:12:39,120-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [5861e77d] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 02:13:39,139-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler7) [61322c25] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 02:14:39,158-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [304bc8c5] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 02:15:39,175-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [3cb1fd7] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 02:16:39,196-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler7) [bb8e62f] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 02:17:39,220-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [29a7a422] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 02:18:39,240-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler7) [68fc0d97] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 02:19:39,259-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [ea9b51a] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 02:20:24,194-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.OvfDataUpdater] (DefaultQuartzScheduler4) [1a577aa] Attempting to update VMs/Templates Ovf. 2017-10-01 02:20:24,197-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler4) [7e37e1ad] Before acquiring and wait lock 'EngineLock:{exclusiveLocks='[aca61eac-ed8a-484b-bbee-3645148f6405=OVF_UPDATE]', sharedLocks=''}' 2017-10-01 02:20:24,197-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler4) [7e37e1ad] Lock-wait acquired to object 'EngineLock:{exclusiveLocks='[aca61eac-ed8a-484b-bbee-3645148f6405=OVF_UPDATE]', sharedLocks=''}' 2017-10-01 02:20:24,199-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler4) [7e37e1ad] Running command: ProcessOvfUpdateForStoragePoolCommand internal: true. Entities affected : ID: aca61eac-ed8a-484b-bbee-3645148f6405 Type: StoragePool 2017-10-01 02:20:24,204-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler4) [7e37e1ad] Attempting to update VM OVFs in Data Center 'EECS-Primary' 2017-10-01 02:20:24,208-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler4) [7e37e1ad] Successfully updated VM OVFs in Data Center 'EECS-Primary' 2017-10-01 02:20:24,208-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler4) [7e37e1ad] Attempting to update template OVFs in Data Center 'EECS-Primary' 2017-10-01 02:20:24,208-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler4) [7e37e1ad] Successfully updated templates OVFs in Data Center 'EECS-Primary' 2017-10-01 02:20:24,208-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler4) [7e37e1ad] Attempting to remove unneeded template/vm OVFs in Data Center 'EECS-Primary' 2017-10-01 02:20:24,210-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler4) [7e37e1ad] Successfully removed unneeded template/vm OVFs in Data Center 'EECS-Primary' 2017-10-01 02:20:24,211-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler4) [7e37e1ad] Lock freed to object 'EngineLock:{exclusiveLocks='[aca61eac-ed8a-484b-bbee-3645148f6405=OVF_UPDATE]', sharedLocks=''}' 2017-10-01 02:20:39,278-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [7380a9ed] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 02:21:39,297-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [1c046e1b] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 02:22:39,321-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [71eff464] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 02:23:39,341-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [67d88a86] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 02:24:39,360-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [5f51109b] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 02:25:39,379-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [2445d83c] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 02:26:39,399-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [2a26dd8a] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 02:27:39,423-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [7430c4af] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 02:28:17,506-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-6-thread-28) [1273bfcf] START, GetFileStatsVDSCommand( GetFileStatsParameters:{runAsync='true', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', ignoreFailoverLimit='false'}), log id: 64b6d4ec 2017-10-01 02:28:18,519-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-6-thread-28) [1273bfcf] FINISH, GetFileStatsVDSCommand, return: {virtio-win-0.1.126.iso={status=0, ctime=1497024605.0, size=155856896}, server2012r2_virtio.iso={status=0, ctime=1497297085.0, size=4977590272}, en_windows_7_professional_with_sp1_x64_dvd_621750.iso={status=0, ctime=1435288543.0, size=3319478272}, virtio-win-0.1.105.iso={status=0, ctime=1435288648.0, size=56584192}}, log id: 64b6d4ec 2017-10-01 02:28:18,523-04 INFO [org.ovirt.engine.core.bll.storage.domain.IsoDomainListSynchronizer] (org.ovirt.thread.pool-6-thread-28) [1273bfcf] Finished automatic refresh process for 'ISO' file type with success, for storage domain id '63d680c7-4c94-4b29-a6cb-f288a128b9bc'. 2017-10-01 02:28:39,445-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [210f81f9] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 02:29:39,467-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [1666104] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 02:30:39,486-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [2e9e1c41] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 02:31:39,505-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [7a7bfa1c] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 02:32:39,529-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [2460f320] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 02:33:39,548-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [76376117] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 02:34:39,568-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [28841609] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 02:35:39,589-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [143dbb74] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 02:36:39,606-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [275de952] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 02:37:39,630-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [3b8abac6] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 02:38:39,650-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [6cf1d876] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 02:39:39,666-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [2065856d] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 02:40:39,685-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler7) [6b203ea0] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 02:41:39,705-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [1a93e626] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 02:42:39,730-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [94befb] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 02:43:39,752-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler7) [6356d406] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 02:44:39,774-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [8e7e4f2] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 02:45:39,794-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [1a26f02a] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 02:46:39,813-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [4cfb1801] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 02:47:39,838-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [2d3c0c9b] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 02:48:39,855-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [299af6b9] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 02:49:39,873-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [220944df] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 02:50:39,894-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [71e0da65] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 02:51:39,913-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [4a7ddac1] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 02:52:39,935-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [1dbdf001] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 02:53:39,954-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler7) [6105c4e4] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 02:54:39,974-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [2246b313] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 02:55:39,991-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [664c7712] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 02:56:40,008-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [5aeb465] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 02:57:40,032-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [7d03ca70] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 02:58:40,051-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [83f30e2] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 02:59:40,072-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [5b2587be] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 03:00:40,091-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [69e45653] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 03:01:40,111-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [5237dac5] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 03:02:40,134-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [1afc4ecb] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 03:03:40,153-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler7) [7263ad44] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 03:04:40,176-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [4e9813b5] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 03:04:58,874-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [7daccede] EVENT_ID: HOST_BOND_SLAVE_STATE_DOWN(612), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Slave p3p1 of bond bond1 on host virt3, changed state to down 2017-10-01 03:05:14,003-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [37f9dbf0] EVENT_ID: HOST_BOND_SLAVE_STATE_UP(611), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Slave p3p1 of bond bond1 on host virt3, changed state to up 2017-10-01 03:05:40,196-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [1310a1] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 03:06:40,213-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [516ae72c] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 03:07:40,237-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [6fb52b75] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 03:08:40,256-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [792f86bf] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 03:09:40,276-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [2fd0cc82] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 03:10:40,295-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler7) [20b0d3fd] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 03:11:40,314-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [15845141] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 03:12:40,338-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [ffa69fc] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 03:13:40,356-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [cfaefeb] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 03:14:40,373-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [6f4c0209] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 03:15:40,390-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [7b68df0c] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 03:16:40,412-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [62eae95] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 03:17:40,438-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [20eaeab] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 03:18:40,457-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [f38fa03] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 03:19:40,477-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [4bb4418f] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 03:20:24,212-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.OvfDataUpdater] (DefaultQuartzScheduler5) [761b7942] Attempting to update VMs/Templates Ovf. 2017-10-01 03:20:24,214-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler5) [6497bae] Before acquiring and wait lock 'EngineLock:{exclusiveLocks='[aca61eac-ed8a-484b-bbee-3645148f6405=OVF_UPDATE]', sharedLocks=''}' 2017-10-01 03:20:24,214-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler5) [6497bae] Lock-wait acquired to object 'EngineLock:{exclusiveLocks='[aca61eac-ed8a-484b-bbee-3645148f6405=OVF_UPDATE]', sharedLocks=''}' 2017-10-01 03:20:24,215-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler5) [6497bae] Running command: ProcessOvfUpdateForStoragePoolCommand internal: true. Entities affected : ID: aca61eac-ed8a-484b-bbee-3645148f6405 Type: StoragePool 2017-10-01 03:20:24,220-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler5) [6497bae] Attempting to update VM OVFs in Data Center 'EECS-Primary' 2017-10-01 03:20:24,225-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler5) [6497bae] Successfully updated VM OVFs in Data Center 'EECS-Primary' 2017-10-01 03:20:24,225-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler5) [6497bae] Attempting to update template OVFs in Data Center 'EECS-Primary' 2017-10-01 03:20:24,225-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler5) [6497bae] Successfully updated templates OVFs in Data Center 'EECS-Primary' 2017-10-01 03:20:24,225-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler5) [6497bae] Attempting to remove unneeded template/vm OVFs in Data Center 'EECS-Primary' 2017-10-01 03:20:24,226-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler5) [6497bae] Successfully removed unneeded template/vm OVFs in Data Center 'EECS-Primary' 2017-10-01 03:20:24,227-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler5) [6497bae] Lock freed to object 'EngineLock:{exclusiveLocks='[aca61eac-ed8a-484b-bbee-3645148f6405=OVF_UPDATE]', sharedLocks=''}' 2017-10-01 03:20:40,497-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [5f3048c7] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 03:21:40,519-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler7) [7f2b0dff] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 03:22:40,540-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler7) [5fd43ce0] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 03:23:40,559-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [17c19a66] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 03:24:40,576-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [2ae6e031] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 03:25:40,596-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [46f4323b] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 03:26:40,616-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler7) [289c89c4] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 03:27:40,639-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [2f6ce412] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 03:28:18,526-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-6-thread-13) [641d373b] START, GetFileStatsVDSCommand( GetFileStatsParameters:{runAsync='true', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', ignoreFailoverLimit='false'}), log id: 5f86b124 2017-10-01 03:28:18,887-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-6-thread-13) [641d373b] FINISH, GetFileStatsVDSCommand, return: {virtio-win-0.1.126.iso={status=0, ctime=1497024605.0, size=155856896}, server2012r2_virtio.iso={status=0, ctime=1497297085.0, size=4977590272}, en_windows_7_professional_with_sp1_x64_dvd_621750.iso={status=0, ctime=1435288543.0, size=3319478272}, virtio-win-0.1.105.iso={status=0, ctime=1435288648.0, size=56584192}}, log id: 5f86b124 2017-10-01 03:28:18,890-04 INFO [org.ovirt.engine.core.bll.storage.domain.IsoDomainListSynchronizer] (org.ovirt.thread.pool-6-thread-13) [641d373b] Finished automatic refresh process for 'ISO' file type with success, for storage domain id '63d680c7-4c94-4b29-a6cb-f288a128b9bc'. 2017-10-01 03:28:40,658-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [73ec49de] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 03:29:40,677-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler7) [7c7c090e] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 03:30:40,693-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler7) [1b0297b1] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 03:31:40,715-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [70d378bb] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 03:32:40,740-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [63f61553] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 03:33:40,761-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [3d8e751e] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 03:34:40,780-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [d1be058] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 03:35:35,000-04 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler10) [3e43ae1e] Start deleteAgedOutAuditLogs 2017-10-01 03:35:35,001-04 INFO [org.ovirt.engine.core.bll.CommandEntityCleanupManager] (DefaultQuartzScheduler2) [5c597418] Start deleteAgedOutCommandEntities 2017-10-01 03:35:35,004-04 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler10) [3e43ae1e] Finished deleteAgedOutAuditLogs 2017-10-01 03:35:35,005-04 INFO [org.ovirt.engine.core.bll.CommandEntityCleanupManager] (DefaultQuartzScheduler2) [5c597418] Finished deleteAgedOutCommandEntities 2017-10-01 03:35:40,799-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler7) [129cc967] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 03:36:40,817-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [51af7f34] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 03:37:40,845-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [409058b2] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 03:38:40,862-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [3af2842f] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 03:39:40,883-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [40448197] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 03:40:40,901-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [6d1ed4d3] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 03:41:40,920-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler7) [304576e6] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 03:42:40,945-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [8d2907] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 03:43:40,965-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [4a987cdf] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 03:44:40,984-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler7) [7a4511b8] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 03:45:41,004-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [66626b6b] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 03:46:41,024-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [466be9c9] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 03:47:41,048-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [46363909] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 03:48:41,070-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [9078400] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 03:49:41,090-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [641ab2a8] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 03:50:41,109-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [128644d5] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 03:51:41,128-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [59183d07] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 03:52:41,152-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [5a0ac725] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 03:53:41,171-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [c749a7c] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 03:54:41,190-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler7) [35bce8fe] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 03:55:41,210-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [2350d7de] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 03:56:41,229-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [23023507] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 03:57:41,253-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler7) [72f882b8] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 03:58:41,274-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [3531706a] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 03:59:41,293-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [298bcbd6] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 04:00:41,312-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [420797] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 04:01:41,330-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [751281a2] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 04:02:41,356-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler7) [7d7bf85c] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 04:03:41,375-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [41b74d00] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 04:04:41,395-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [510ce141] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 04:05:41,415-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [60f2f4a] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 04:06:41,435-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [5748321d] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 04:07:41,457-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [aca4154] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 04:08:41,473-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [470c3237] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 04:09:41,491-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [43a6e441] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 04:10:41,511-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [5d220b46] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 04:11:41,530-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [4a996785] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 04:12:41,555-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [127966a6] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 04:13:41,576-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [22dde49b] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 04:14:41,596-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler7) [52565666] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 04:15:41,616-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [5965f0ee] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 04:16:41,636-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [5f0a3c72] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 04:17:41,660-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [5c918656] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 04:18:41,681-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [70ea34de] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 04:19:41,701-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [3a049d8e] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 04:20:24,228-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.OvfDataUpdater] (DefaultQuartzScheduler8) [366ba746] Attempting to update VMs/Templates Ovf. 2017-10-01 04:20:24,233-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler8) [3c40d875] Before acquiring and wait lock 'EngineLock:{exclusiveLocks='[aca61eac-ed8a-484b-bbee-3645148f6405=OVF_UPDATE]', sharedLocks=''}' 2017-10-01 04:20:24,233-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler8) [3c40d875] Lock-wait acquired to object 'EngineLock:{exclusiveLocks='[aca61eac-ed8a-484b-bbee-3645148f6405=OVF_UPDATE]', sharedLocks=''}' 2017-10-01 04:20:24,234-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler8) [3c40d875] Running command: ProcessOvfUpdateForStoragePoolCommand internal: true. Entities affected : ID: aca61eac-ed8a-484b-bbee-3645148f6405 Type: StoragePool 2017-10-01 04:20:24,297-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler8) [3c40d875] Attempting to update VM OVFs in Data Center 'EECS-Primary' 2017-10-01 04:20:24,298-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler8) [3c40d875] Successfully updated VM OVFs in Data Center 'EECS-Primary' 2017-10-01 04:20:24,298-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler8) [3c40d875] Attempting to update template OVFs in Data Center 'EECS-Primary' 2017-10-01 04:20:24,298-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler8) [3c40d875] Successfully updated templates OVFs in Data Center 'EECS-Primary' 2017-10-01 04:20:24,298-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler8) [3c40d875] Attempting to remove unneeded template/vm OVFs in Data Center 'EECS-Primary' 2017-10-01 04:20:24,299-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler8) [3c40d875] Successfully removed unneeded template/vm OVFs in Data Center 'EECS-Primary' 2017-10-01 04:20:24,301-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler8) [3c40d875] Lock freed to object 'EngineLock:{exclusiveLocks='[aca61eac-ed8a-484b-bbee-3645148f6405=OVF_UPDATE]', sharedLocks=''}' 2017-10-01 04:20:41,720-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [68aeb35f] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 04:21:41,740-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [5e74556a] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 04:22:41,764-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler7) [708f594a] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 04:23:41,782-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [8102c22] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 04:24:41,803-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [799aad8e] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 04:25:41,823-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [7f9ff17a] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 04:26:41,842-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [2e3a76d8] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 04:27:23,227-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [4099f475] EVENT_ID: HOST_BOND_SLAVE_STATE_DOWN(612), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Slave p3p1 of bond bond1 on host virt3, changed state to down 2017-10-01 04:27:38,363-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [6ee94ff1] EVENT_ID: HOST_BOND_SLAVE_STATE_UP(611), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Slave p3p1 of bond bond1 on host virt3, changed state to up 2017-10-01 04:27:41,866-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [57546fb5] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 04:28:18,892-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-6-thread-31) [30811640] START, GetFileStatsVDSCommand( GetFileStatsParameters:{runAsync='true', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', ignoreFailoverLimit='false'}), log id: 53a0ed7a 2017-10-01 04:28:21,047-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-6-thread-31) [30811640] FINISH, GetFileStatsVDSCommand, return: {virtio-win-0.1.126.iso={status=0, ctime=1497024605.0, size=155856896}, server2012r2_virtio.iso={status=0, ctime=1497297085.0, size=4977590272}, en_windows_7_professional_with_sp1_x64_dvd_621750.iso={status=0, ctime=1435288543.0, size=3319478272}, virtio-win-0.1.105.iso={status=0, ctime=1435288648.0, size=56584192}}, log id: 53a0ed7a 2017-10-01 04:28:21,051-04 INFO [org.ovirt.engine.core.bll.storage.domain.IsoDomainListSynchronizer] (org.ovirt.thread.pool-6-thread-31) [30811640] Finished automatic refresh process for 'ISO' file type with success, for storage domain id '63d680c7-4c94-4b29-a6cb-f288a128b9bc'. 2017-10-01 04:28:41,885-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [565fe3ec] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 04:29:41,903-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [35c21c09] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 04:30:41,923-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [4ec1f16c] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 04:31:41,940-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [5477451d] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 04:32:41,965-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [60cd162c] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 04:33:41,987-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [1fb378b0] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 04:34:42,006-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [d67f916] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 04:35:42,026-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [5b2c9dd3] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 04:36:42,047-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [1e368d0f] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 04:37:42,070-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler7) [701ba4fc] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 04:38:42,087-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler7) [61d4212d] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 04:39:42,106-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [6e72eba5] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 04:40:42,126-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler7) [2ef6566f] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 04:41:42,145-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [1c6d1409] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 04:42:42,172-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [67a5d12a] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 04:43:42,192-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [49316119] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 04:44:42,211-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [4bba3958] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 04:45:42,230-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [3e0e7b85] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 04:46:42,249-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [44f11ff6] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 04:47:42,273-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [7e596d5] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 04:48:42,291-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [17d3e8d3] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 04:49:42,308-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [1340be00] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 04:50:42,333-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [4c240172] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 04:51:42,353-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [3fba80df] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 04:52:42,378-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [7604ea68] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 04:53:42,398-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [5e83f6ef] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 04:54:42,417-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [654135e2] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 04:55:42,437-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [4f9b8529] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 04:56:42,456-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [19870e0e] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 04:57:42,478-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [45d86102] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 04:58:42,498-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [3e64c855] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 04:59:42,517-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [19f744e] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 05:00:42,537-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [36e5fa3d] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 05:01:42,558-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [573feb52] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 05:02:42,582-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [2f8876fe] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 05:03:42,602-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [35dba3a5] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 05:04:42,622-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [6b738a5] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 05:05:42,641-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler7) [62a84b14] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 05:06:42,662-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [3347ff51] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 05:07:42,689-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [6a28b7e0] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 05:08:42,711-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [14456411] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 05:09:42,728-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [233d7a3a] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 05:10:42,748-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [21bbb2c6] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 05:11:42,769-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [20b5545d] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 05:12:42,794-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [7c3e9487] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 05:13:42,813-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [37bdc138] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 05:14:42,832-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [26d7d05] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 05:15:42,852-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [171915f0] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 05:16:42,872-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [acf7b90] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 05:17:42,895-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [4d97c60] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 05:18:42,914-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [320a2bba] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 05:19:42,933-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler7) [60d25571] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 05:20:24,302-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.OvfDataUpdater] (DefaultQuartzScheduler4) [693256a1] Attempting to update VMs/Templates Ovf. 2017-10-01 05:20:24,306-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler4) [19fca8bb] Before acquiring and wait lock 'EngineLock:{exclusiveLocks='[aca61eac-ed8a-484b-bbee-3645148f6405=OVF_UPDATE]', sharedLocks=''}' 2017-10-01 05:20:24,306-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler4) [19fca8bb] Lock-wait acquired to object 'EngineLock:{exclusiveLocks='[aca61eac-ed8a-484b-bbee-3645148f6405=OVF_UPDATE]', sharedLocks=''}' 2017-10-01 05:20:24,307-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler4) [19fca8bb] Running command: ProcessOvfUpdateForStoragePoolCommand internal: true. Entities affected : ID: aca61eac-ed8a-484b-bbee-3645148f6405 Type: StoragePool 2017-10-01 05:20:24,367-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler4) [19fca8bb] Attempting to update VM OVFs in Data Center 'EECS-Primary' 2017-10-01 05:20:24,368-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler4) [19fca8bb] Successfully updated VM OVFs in Data Center 'EECS-Primary' 2017-10-01 05:20:24,368-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler4) [19fca8bb] Attempting to update template OVFs in Data Center 'EECS-Primary' 2017-10-01 05:20:24,368-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler4) [19fca8bb] Successfully updated templates OVFs in Data Center 'EECS-Primary' 2017-10-01 05:20:24,368-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler4) [19fca8bb] Attempting to remove unneeded template/vm OVFs in Data Center 'EECS-Primary' 2017-10-01 05:20:24,370-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler4) [19fca8bb] Successfully removed unneeded template/vm OVFs in Data Center 'EECS-Primary' 2017-10-01 05:20:24,371-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler4) [19fca8bb] Lock freed to object 'EngineLock:{exclusiveLocks='[aca61eac-ed8a-484b-bbee-3645148f6405=OVF_UPDATE]', sharedLocks=''}' 2017-10-01 05:20:42,950-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler7) [28d90555] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 05:21:42,972-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [3c7b9808] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 05:22:42,996-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [45d6731] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 05:23:43,018-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [2c9a35f4] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 05:24:43,038-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [267c57d4] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 05:25:43,058-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [2876c05c] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 05:26:43,078-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [32c4e3b7] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 05:27:43,103-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [3e0dfd94] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 05:28:21,052-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-6-thread-4) [1d486911] START, GetFileStatsVDSCommand( GetFileStatsParameters:{runAsync='true', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', ignoreFailoverLimit='false'}), log id: 6419eb7d 2017-10-01 05:28:21,269-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-6-thread-4) [1d486911] FINISH, GetFileStatsVDSCommand, return: {virtio-win-0.1.126.iso={status=0, ctime=1497024605.0, size=155856896}, server2012r2_virtio.iso={status=0, ctime=1497297085.0, size=4977590272}, en_windows_7_professional_with_sp1_x64_dvd_621750.iso={status=0, ctime=1435288543.0, size=3319478272}, virtio-win-0.1.105.iso={status=0, ctime=1435288648.0, size=56584192}}, log id: 6419eb7d 2017-10-01 05:28:21,272-04 INFO [org.ovirt.engine.core.bll.storage.domain.IsoDomainListSynchronizer] (org.ovirt.thread.pool-6-thread-4) [1d486911] Finished automatic refresh process for 'ISO' file type with success, for storage domain id '63d680c7-4c94-4b29-a6cb-f288a128b9bc'. 2017-10-01 05:28:43,122-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [735f3df0] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 05:29:43,141-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [2c0f2177] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 05:30:43,159-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [2973fcc3] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 05:31:43,179-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [2f6d734c] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 05:32:43,205-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [40082947] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 05:33:43,225-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [1f265eb0] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 05:34:43,244-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [79e9c30f] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 05:35:43,263-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [30a8e21f] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 05:36:43,282-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [63d0ba8b] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 05:37:43,307-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [7f7d406d] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 05:38:43,330-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler7) [5cd4a69a] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 05:39:43,349-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [7cdb9ca8] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 05:40:43,373-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [4cb1a0a] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 05:41:43,392-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [7d665ba4] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 05:42:43,416-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [7bd0c42d] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 05:43:43,435-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [6775de76] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 05:44:43,454-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [13fe6f77] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 05:45:43,471-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [222ffbdc] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 05:46:43,492-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [26d0e1d] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 05:47:43,517-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [19f25e22] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 05:48:43,536-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [15738ccb] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 05:49:43,555-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler7) [537a2e4a] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 05:50:43,576-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [5438f680] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 05:51:43,595-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [10479f2a] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 05:52:43,621-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [897d3f7] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 05:53:43,638-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [88ce775] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 05:54:43,657-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [49eb8fa5] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 05:55:43,677-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [193c7ce7] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 05:56:43,697-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [1e3eb9ba] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 05:57:43,721-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [1368d495] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 05:58:43,740-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [787c8d24] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 05:59:43,757-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [306f6f21] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 06:00:43,780-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler7) [6f67c0a6] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 06:01:43,800-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [17cbd52b] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 06:02:43,826-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [6713331b] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 06:03:43,847-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [1412edda] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 06:04:43,866-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [308c613c] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 06:05:43,885-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [644b9da] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 06:06:43,906-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [3d66170c] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 06:07:43,930-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [2562f74f] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 06:08:43,950-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [56de2289] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 06:09:43,973-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [d031a5d] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 06:10:43,993-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [1c7e5c5d] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 06:11:44,013-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [6eaf115a] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 06:12:44,037-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [45cc029a] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 06:13:44,057-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [7de4fce7] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 06:14:44,077-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [44e354b1] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 06:15:44,096-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [58f77bfd] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 06:16:44,118-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [32768486] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 06:17:44,146-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [5c491839] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 06:18:44,166-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [72f62c7c] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 06:19:20,454-04 INFO [org.ovirt.engine.core.bll.EngineBackupAwarenessManager] (DefaultQuartzScheduler8) [465941d5] Backup check started. 2017-10-01 06:19:20,457-04 INFO [org.ovirt.engine.core.bll.EngineBackupAwarenessManager] (DefaultQuartzScheduler8) [465941d5] Backup check completed. 2017-10-01 06:19:44,186-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [320d61b5] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 06:20:24,371-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.OvfDataUpdater] (DefaultQuartzScheduler9) [3b2b33b4] Attempting to update VMs/Templates Ovf. 2017-10-01 06:20:24,374-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler9) [45b56d1a] Before acquiring and wait lock 'EngineLock:{exclusiveLocks='[aca61eac-ed8a-484b-bbee-3645148f6405=OVF_UPDATE]', sharedLocks=''}' 2017-10-01 06:20:24,374-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler9) [45b56d1a] Lock-wait acquired to object 'EngineLock:{exclusiveLocks='[aca61eac-ed8a-484b-bbee-3645148f6405=OVF_UPDATE]', sharedLocks=''}' 2017-10-01 06:20:24,375-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler9) [45b56d1a] Running command: ProcessOvfUpdateForStoragePoolCommand internal: true. Entities affected : ID: aca61eac-ed8a-484b-bbee-3645148f6405 Type: StoragePool 2017-10-01 06:20:24,434-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler9) [45b56d1a] Attempting to update VM OVFs in Data Center 'EECS-Primary' 2017-10-01 06:20:24,438-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler9) [45b56d1a] Successfully updated VM OVFs in Data Center 'EECS-Primary' 2017-10-01 06:20:24,438-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler9) [45b56d1a] Attempting to update template OVFs in Data Center 'EECS-Primary' 2017-10-01 06:20:24,439-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler9) [45b56d1a] Successfully updated templates OVFs in Data Center 'EECS-Primary' 2017-10-01 06:20:24,439-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler9) [45b56d1a] Attempting to remove unneeded template/vm OVFs in Data Center 'EECS-Primary' 2017-10-01 06:20:24,440-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler9) [45b56d1a] Successfully removed unneeded template/vm OVFs in Data Center 'EECS-Primary' 2017-10-01 06:20:24,441-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler9) [45b56d1a] Lock freed to object 'EngineLock:{exclusiveLocks='[aca61eac-ed8a-484b-bbee-3645148f6405=OVF_UPDATE]', sharedLocks=''}' 2017-10-01 06:20:44,207-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [3819c69a] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 06:21:44,232-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [6e1c0376] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 06:22:44,258-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [3d78b682] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 06:23:44,300-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler7) [257b71a] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 06:24:44,319-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [c9bf866] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 06:25:44,337-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [55b77199] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 06:26:44,360-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [206f3d2d] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 06:27:44,385-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [e3b1f0f] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 06:28:21,274-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-6-thread-9) [68f62dfe] START, GetFileStatsVDSCommand( GetFileStatsParameters:{runAsync='true', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', ignoreFailoverLimit='false'}), log id: 3c0351ed 2017-10-01 06:28:21,649-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-6-thread-9) [68f62dfe] FINISH, GetFileStatsVDSCommand, return: {virtio-win-0.1.126.iso={status=0, ctime=1497024605.0, size=155856896}, server2012r2_virtio.iso={status=0, ctime=1497297085.0, size=4977590272}, en_windows_7_professional_with_sp1_x64_dvd_621750.iso={status=0, ctime=1435288543.0, size=3319478272}, virtio-win-0.1.105.iso={status=0, ctime=1435288648.0, size=56584192}}, log id: 3c0351ed 2017-10-01 06:28:21,653-04 INFO [org.ovirt.engine.core.bll.storage.domain.IsoDomainListSynchronizer] (org.ovirt.thread.pool-6-thread-9) [68f62dfe] Finished automatic refresh process for 'ISO' file type with success, for storage domain id '63d680c7-4c94-4b29-a6cb-f288a128b9bc'. 2017-10-01 06:28:44,409-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [5990506b] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 06:29:44,428-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [40dbb540] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 06:30:44,447-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [f4e6c81] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 06:31:44,467-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [39e330ff] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 06:32:44,497-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [3bf99a6d] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 06:33:44,518-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [64f10ce7] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 06:34:44,537-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [628506d4] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 06:35:44,556-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler7) [60ef2912] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 06:36:44,575-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [4d8b9383] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 06:37:44,601-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [1a06764d] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 06:38:44,623-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [5c1dadd4] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 06:39:44,642-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [62549be4] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 06:40:44,664-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [5ee9d10a] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 06:41:44,683-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [1595ee6a] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 06:42:44,707-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [62c43eda] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 06:43:44,726-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [59055864] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 06:44:44,745-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler7) [2bcfab7c] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 06:45:44,766-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [1c59e678] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 06:46:44,785-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [14bf9afd] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 06:47:44,810-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [1c869f7a] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 06:48:44,829-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [55555bf4] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 06:49:44,851-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [27bf4e5] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 06:50:44,870-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [5516cf1b] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 06:51:44,890-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [1389a51a] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 06:52:44,915-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [6101ffa2] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 06:53:44,935-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [151f6ffd] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 06:54:44,953-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [3e997e0] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 06:55:44,973-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [78e53f04] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 06:56:44,992-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [7618cef] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 06:57:45,021-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [3d6db955] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 06:58:45,040-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [6f318854] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 06:59:45,060-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [28488355] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 07:00:45,079-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [78fe7b85] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 07:01:45,099-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [20c107b3] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 07:02:45,122-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [7db83a44] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 07:03:45,142-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [1f42fb6a] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 07:04:45,164-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [2a4786ef] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 07:05:45,184-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler7) [2f2dfde6] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 07:06:45,202-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [4d2f2631] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 07:07:45,226-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [7cfc4bb8] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 07:08:45,247-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [5f65349d] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 07:09:45,267-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [645a67ba] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 07:10:45,286-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [50f3e5e6] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 07:11:45,307-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [15cb40ed] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 07:12:45,332-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [619721d4] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 07:13:45,353-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [26ed4d19] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 07:14:45,372-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [159818e8] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 07:15:45,391-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [7d9d9d2a] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 07:16:45,410-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [7d67f826] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 07:17:45,435-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [792d5bcd] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 07:18:45,455-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [2f15dcc8] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 07:19:45,474-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [15fe61d4] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 07:20:24,442-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.OvfDataUpdater] (DefaultQuartzScheduler9) [6aabb07] Attempting to update VMs/Templates Ovf. 2017-10-01 07:20:24,444-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler9) [7ba5f49d] Before acquiring and wait lock 'EngineLock:{exclusiveLocks='[aca61eac-ed8a-484b-bbee-3645148f6405=OVF_UPDATE]', sharedLocks=''}' 2017-10-01 07:20:24,444-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler9) [7ba5f49d] Lock-wait acquired to object 'EngineLock:{exclusiveLocks='[aca61eac-ed8a-484b-bbee-3645148f6405=OVF_UPDATE]', sharedLocks=''}' 2017-10-01 07:20:24,446-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler9) [7ba5f49d] Running command: ProcessOvfUpdateForStoragePoolCommand internal: true. Entities affected : ID: aca61eac-ed8a-484b-bbee-3645148f6405 Type: StoragePool 2017-10-01 07:20:24,451-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler9) [7ba5f49d] Attempting to update VM OVFs in Data Center 'EECS-Primary' 2017-10-01 07:20:24,456-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler9) [7ba5f49d] Successfully updated VM OVFs in Data Center 'EECS-Primary' 2017-10-01 07:20:24,456-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler9) [7ba5f49d] Attempting to update template OVFs in Data Center 'EECS-Primary' 2017-10-01 07:20:24,457-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler9) [7ba5f49d] Successfully updated templates OVFs in Data Center 'EECS-Primary' 2017-10-01 07:20:24,457-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler9) [7ba5f49d] Attempting to remove unneeded template/vm OVFs in Data Center 'EECS-Primary' 2017-10-01 07:20:24,458-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler9) [7ba5f49d] Successfully removed unneeded template/vm OVFs in Data Center 'EECS-Primary' 2017-10-01 07:20:24,459-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler9) [7ba5f49d] Lock freed to object 'EngineLock:{exclusiveLocks='[aca61eac-ed8a-484b-bbee-3645148f6405=OVF_UPDATE]', sharedLocks=''}' 2017-10-01 07:20:45,493-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [5c5f84f1] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 07:21:45,511-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [2c087ce5] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 07:22:45,536-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [4886d27d] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 07:23:45,555-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [58d44b6a] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 07:24:45,574-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [4c491b81] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 07:25:45,594-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [2fa77e67] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 07:26:45,614-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [56c14ee2] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 07:27:45,638-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [23e3797d] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 07:28:21,655-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-6-thread-46) [38274daf] START, GetFileStatsVDSCommand( GetFileStatsParameters:{runAsync='true', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', ignoreFailoverLimit='false'}), log id: 1e468f5f 2017-10-01 07:28:22,669-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-6-thread-46) [38274daf] FINISH, GetFileStatsVDSCommand, return: {virtio-win-0.1.126.iso={status=0, ctime=1497024605.0, size=155856896}, server2012r2_virtio.iso={status=0, ctime=1497297085.0, size=4977590272}, en_windows_7_professional_with_sp1_x64_dvd_621750.iso={status=0, ctime=1435288543.0, size=3319478272}, virtio-win-0.1.105.iso={status=0, ctime=1435288648.0, size=56584192}}, log id: 1e468f5f 2017-10-01 07:28:22,672-04 INFO [org.ovirt.engine.core.bll.storage.domain.IsoDomainListSynchronizer] (org.ovirt.thread.pool-6-thread-46) [38274daf] Finished automatic refresh process for 'ISO' file type with success, for storage domain id '63d680c7-4c94-4b29-a6cb-f288a128b9bc'. 2017-10-01 07:28:45,662-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [26a63e01] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 07:29:45,682-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [6bc9f929] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 07:30:45,702-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [1885d733] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 07:31:45,720-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [1f175c02] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 07:32:45,744-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [6b0d99b1] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 07:33:45,763-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [41a1c1f6] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 07:34:45,782-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [6f65a1eb] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 07:35:45,803-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler7) [309f7eae] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 07:36:45,823-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [29db114f] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 07:37:45,848-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [545e4593] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 07:38:45,869-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [2d8352de] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 07:39:45,889-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [7f7003b6] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 07:40:45,906-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [131bc4e5] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 07:41:45,927-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [4d3d1eb1] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 07:42:45,954-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [463adf2e] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 07:43:45,971-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [44da49b9] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 07:44:45,990-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [3808d50d] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 07:45:46,009-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [6bb0e51f] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 07:46:46,029-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [5452d40a] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 07:47:46,053-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [56f09881] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 07:48:46,072-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [191abcb6] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 07:49:46,088-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [6de414a0] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 07:50:46,107-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [f3a31b5] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 07:51:46,126-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [7ce36458] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 07:52:46,151-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [2030b101] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 07:53:46,170-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [298421be] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 07:54:46,190-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [72a72f68] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 07:55:46,211-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [1ed4902a] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 07:56:46,230-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler7) [11b749f5] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 07:57:46,254-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [2d64704] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 07:58:46,273-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [8ed82ee] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 07:59:46,295-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [4a6c0575] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 08:00:46,314-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [68203e88] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 08:01:46,334-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler7) [60bac37] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 08:02:46,358-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [b759186] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 08:03:46,378-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [40a1382e] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 08:04:46,400-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [7aebdb3f] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 08:05:46,418-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [6e879556] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 08:06:46,438-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [2c935399] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 08:07:46,463-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [30fcc16] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 08:08:46,484-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [5dcfa32f] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 08:09:46,504-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [78690651] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 08:10:46,524-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [20b231b9] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 08:11:46,542-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [2805baa2] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 08:12:46,567-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [2b91c0fe] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 08:13:46,589-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [478265ec] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 08:14:46,608-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [6a49a5e6] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 08:15:46,625-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [2960d8c1] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 08:16:46,645-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [4c976ee8] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 08:17:46,670-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [268a0296] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 08:18:46,689-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [ad75ce4] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 08:19:46,709-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [22c7e466] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 08:20:24,461-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.OvfDataUpdater] (DefaultQuartzScheduler9) [68104bb5] Attempting to update VMs/Templates Ovf. 2017-10-01 08:20:24,464-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler9) [44f524b7] Before acquiring and wait lock 'EngineLock:{exclusiveLocks='[aca61eac-ed8a-484b-bbee-3645148f6405=OVF_UPDATE]', sharedLocks=''}' 2017-10-01 08:20:24,464-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler9) [44f524b7] Lock-wait acquired to object 'EngineLock:{exclusiveLocks='[aca61eac-ed8a-484b-bbee-3645148f6405=OVF_UPDATE]', sharedLocks=''}' 2017-10-01 08:20:24,465-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler9) [44f524b7] Running command: ProcessOvfUpdateForStoragePoolCommand internal: true. Entities affected : ID: aca61eac-ed8a-484b-bbee-3645148f6405 Type: StoragePool 2017-10-01 08:20:24,470-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler9) [44f524b7] Attempting to update VM OVFs in Data Center 'EECS-Primary' 2017-10-01 08:20:24,471-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler9) [44f524b7] Successfully updated VM OVFs in Data Center 'EECS-Primary' 2017-10-01 08:20:24,471-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler9) [44f524b7] Attempting to update template OVFs in Data Center 'EECS-Primary' 2017-10-01 08:20:24,471-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler9) [44f524b7] Successfully updated templates OVFs in Data Center 'EECS-Primary' 2017-10-01 08:20:24,471-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler9) [44f524b7] Attempting to remove unneeded template/vm OVFs in Data Center 'EECS-Primary' 2017-10-01 08:20:24,472-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler9) [44f524b7] Successfully removed unneeded template/vm OVFs in Data Center 'EECS-Primary' 2017-10-01 08:20:24,473-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler9) [44f524b7] Lock freed to object 'EngineLock:{exclusiveLocks='[aca61eac-ed8a-484b-bbee-3645148f6405=OVF_UPDATE]', sharedLocks=''}' 2017-10-01 08:20:46,729-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [4ba89885] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 08:21:46,749-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [15a62b00] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 08:22:46,774-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [31ecba1a] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 08:23:46,794-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [39b987ff] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 08:24:46,814-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [7d8006b5] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 08:25:46,831-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [3d318285] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 08:26:46,851-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [2f99c476] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 08:27:46,875-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [6b31ca0f] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 08:28:22,674-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-6-thread-28) [59d4db48] START, GetFileStatsVDSCommand( GetFileStatsParameters:{runAsync='true', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', ignoreFailoverLimit='false'}), log id: 2295ccd5 2017-10-01 08:28:23,687-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-6-thread-28) [59d4db48] FINISH, GetFileStatsVDSCommand, return: {virtio-win-0.1.126.iso={status=0, ctime=1497024605.0, size=155856896}, server2012r2_virtio.iso={status=0, ctime=1497297085.0, size=4977590272}, en_windows_7_professional_with_sp1_x64_dvd_621750.iso={status=0, ctime=1435288543.0, size=3319478272}, virtio-win-0.1.105.iso={status=0, ctime=1435288648.0, size=56584192}}, log id: 2295ccd5 2017-10-01 08:28:23,690-04 INFO [org.ovirt.engine.core.bll.storage.domain.IsoDomainListSynchronizer] (org.ovirt.thread.pool-6-thread-28) [59d4db48] Finished automatic refresh process for 'ISO' file type with success, for storage domain id '63d680c7-4c94-4b29-a6cb-f288a128b9bc'. 2017-10-01 08:28:46,894-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [4822714a] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 08:29:46,916-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [5dd1d35] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 08:30:46,939-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [3b8019a0] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 08:31:46,958-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler7) [5357295f] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 08:32:46,983-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [48cb2d1] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 08:33:47,003-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [19fd8f71] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 08:34:47,023-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [67fd2878] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 08:35:47,042-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [703b9eb4] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 08:36:47,063-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [23363d47] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 08:37:47,086-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [55ada230] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 08:38:47,106-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler7) [6733a6ca] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 08:39:47,125-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [4e22264c] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 08:40:47,142-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [38d58885] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 08:41:47,161-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [483f153c] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 08:42:47,183-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [2e83374] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 08:43:47,203-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [3fb2fb48] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 08:44:47,222-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [2e12918e] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 08:45:47,241-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [4de3c517] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 08:46:47,266-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [60afaa1b] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 08:47:47,295-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [3225d2dc] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 08:48:47,314-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [150bc346] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 08:49:47,330-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [7b0ca50d] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 08:50:47,350-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [2fb9c697] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 08:51:47,369-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [2b5a348f] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 08:52:47,390-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [719a337b] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 08:53:47,410-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [37ca1cab] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 08:54:47,430-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [7973709b] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 08:55:47,449-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [3c7a018] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 08:56:47,467-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [62b3b0df] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 08:57:47,492-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [7c403848] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 08:58:47,512-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [55d8886a] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 08:59:47,532-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [58e197e3] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 09:00:47,551-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [650bb3c7] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 09:01:47,574-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [63f3a426] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 09:02:47,599-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler7) [78547e07] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 09:03:47,618-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [72c70ec1] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 09:04:47,637-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [43990818] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 09:05:47,655-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [67b999d1] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 09:06:47,674-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [65f43131] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 09:07:47,698-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [75205cc0] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 09:08:47,717-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [35865941] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 09:09:47,736-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [180dff33] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 09:10:47,757-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [7b6b7863] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 09:11:47,777-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [29134a5d] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 09:12:47,801-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [ac8df4c] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 09:13:47,820-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [3660d1df] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 09:14:47,839-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [6521bba] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 09:15:47,858-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [14d292ef] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 09:16:47,874-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [69087751] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 09:17:47,900-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [d97b7f8] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 09:18:47,923-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [5d770010] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 09:19:47,943-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler7) [67d7e72b] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 09:20:24,474-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.OvfDataUpdater] (DefaultQuartzScheduler1) [4eda79ff] Attempting to update VMs/Templates Ovf. 2017-10-01 09:20:24,477-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler1) [307f7209] Before acquiring and wait lock 'EngineLock:{exclusiveLocks='[aca61eac-ed8a-484b-bbee-3645148f6405=OVF_UPDATE]', sharedLocks=''}' 2017-10-01 09:20:24,477-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler1) [307f7209] Lock-wait acquired to object 'EngineLock:{exclusiveLocks='[aca61eac-ed8a-484b-bbee-3645148f6405=OVF_UPDATE]', sharedLocks=''}' 2017-10-01 09:20:24,478-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler1) [307f7209] Running command: ProcessOvfUpdateForStoragePoolCommand internal: true. Entities affected : ID: aca61eac-ed8a-484b-bbee-3645148f6405 Type: StoragePool 2017-10-01 09:20:24,483-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler1) [307f7209] Attempting to update VM OVFs in Data Center 'EECS-Primary' 2017-10-01 09:20:24,487-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler1) [307f7209] Successfully updated VM OVFs in Data Center 'EECS-Primary' 2017-10-01 09:20:24,487-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler1) [307f7209] Attempting to update template OVFs in Data Center 'EECS-Primary' 2017-10-01 09:20:24,488-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler1) [307f7209] Successfully updated templates OVFs in Data Center 'EECS-Primary' 2017-10-01 09:20:24,488-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler1) [307f7209] Attempting to remove unneeded template/vm OVFs in Data Center 'EECS-Primary' 2017-10-01 09:20:24,489-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler1) [307f7209] Successfully removed unneeded template/vm OVFs in Data Center 'EECS-Primary' 2017-10-01 09:20:24,490-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler1) [307f7209] Lock freed to object 'EngineLock:{exclusiveLocks='[aca61eac-ed8a-484b-bbee-3645148f6405=OVF_UPDATE]', sharedLocks=''}' 2017-10-01 09:20:47,962-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [3962c113] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 09:21:47,981-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [7a41535f] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 09:22:48,005-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [6a240eaa] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 09:23:48,024-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [32cf15d] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 09:24:48,044-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [141a61b5] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 09:25:48,061-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [77d7cfde] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 09:26:48,080-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [6cdbc1f] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 09:27:48,103-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [4a509fcf] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 09:28:23,692-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-6-thread-34) [367a6f5c] START, GetFileStatsVDSCommand( GetFileStatsParameters:{runAsync='true', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', ignoreFailoverLimit='false'}), log id: 7914422d 2017-10-01 09:28:24,203-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-6-thread-34) [367a6f5c] FINISH, GetFileStatsVDSCommand, return: {virtio-win-0.1.126.iso={status=0, ctime=1497024605.0, size=155856896}, server2012r2_virtio.iso={status=0, ctime=1497297085.0, size=4977590272}, en_windows_7_professional_with_sp1_x64_dvd_621750.iso={status=0, ctime=1435288543.0, size=3319478272}, virtio-win-0.1.105.iso={status=0, ctime=1435288648.0, size=56584192}}, log id: 7914422d 2017-10-01 09:28:24,206-04 INFO [org.ovirt.engine.core.bll.storage.domain.IsoDomainListSynchronizer] (org.ovirt.thread.pool-6-thread-34) [367a6f5c] Finished automatic refresh process for 'ISO' file type with success, for storage domain id '63d680c7-4c94-4b29-a6cb-f288a128b9bc'. 2017-10-01 09:28:48,121-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [40a1d61b] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 09:29:48,140-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [73f47855] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 09:30:48,162-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [b140210] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 09:31:48,181-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [335fcfee] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 09:32:48,209-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [59635132] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 09:33:48,228-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [1c0b0fb0] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 09:34:48,247-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [5fe70bb2] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 09:35:48,267-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [4f41bda6] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 09:36:48,287-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [740740ae] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 09:37:48,311-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [3a7b53dd] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 09:38:48,327-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [5c494780] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 09:39:48,348-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [396e3dc5] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 09:40:48,368-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [3a42660] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 09:41:48,387-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [e434594] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 09:42:48,411-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [5a57d39f] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 09:43:48,431-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [5219c8c3] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 09:44:48,452-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [6283332] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 09:45:48,472-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [6abd3bcb] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 09:46:48,491-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [27120319] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 09:47:48,515-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [53b3e09d] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 09:48:48,536-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [2d3dbb5c] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 09:49:48,559-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [68fd2dcc] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 09:50:48,580-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [395aad9a] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 09:51:48,599-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [243692fb] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 09:52:48,621-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [6f0a6d11] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 09:53:48,640-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [567868a3] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 09:54:48,659-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [4785b1c1] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 09:55:48,678-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [6ac82d03] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 09:56:48,694-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [6260a0d3] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 09:57:48,718-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [f32f13b] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 09:58:48,737-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [703d5344] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 09:59:48,757-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [6807e544] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 10:00:48,777-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [67ee58e7] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 10:01:48,796-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [1e8df7df] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 10:02:48,818-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [57b29c7f] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 10:03:48,840-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [74e5562f] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 10:04:48,861-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [27a2e08c] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 10:05:48,880-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [1c2bb3a8] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 10:06:48,899-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [2afe1a60] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 10:07:48,924-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [631580e4] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 10:08:48,943-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [392ae3d4] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 10:09:48,963-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler7) [305f99c9] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 10:10:48,982-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [510e4e65] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 10:11:49,001-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [399a8b89] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 10:12:49,024-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [28ae7823] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 10:13:49,043-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [4b2dc4a5] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 10:14:49,064-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [175fb94b] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 10:15:49,084-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [16391fb6] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 10:16:49,104-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [347dcbe5] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 10:17:49,128-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [41570fa2] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 10:18:49,147-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [5bfc4832] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 10:19:49,169-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [2c18d354] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 10:20:24,490-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.OvfDataUpdater] (DefaultQuartzScheduler3) [eb8217d] Attempting to update VMs/Templates Ovf. 2017-10-01 10:20:24,493-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler3) [341fa7e2] Before acquiring and wait lock 'EngineLock:{exclusiveLocks='[aca61eac-ed8a-484b-bbee-3645148f6405=OVF_UPDATE]', sharedLocks=''}' 2017-10-01 10:20:24,493-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler3) [341fa7e2] Lock-wait acquired to object 'EngineLock:{exclusiveLocks='[aca61eac-ed8a-484b-bbee-3645148f6405=OVF_UPDATE]', sharedLocks=''}' 2017-10-01 10:20:24,494-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler3) [341fa7e2] Running command: ProcessOvfUpdateForStoragePoolCommand internal: true. Entities affected : ID: aca61eac-ed8a-484b-bbee-3645148f6405 Type: StoragePool 2017-10-01 10:20:24,560-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler3) [341fa7e2] Attempting to update VM OVFs in Data Center 'EECS-Primary' 2017-10-01 10:20:24,565-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler3) [341fa7e2] Successfully updated VM OVFs in Data Center 'EECS-Primary' 2017-10-01 10:20:24,565-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler3) [341fa7e2] Attempting to update template OVFs in Data Center 'EECS-Primary' 2017-10-01 10:20:24,565-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler3) [341fa7e2] Successfully updated templates OVFs in Data Center 'EECS-Primary' 2017-10-01 10:20:24,565-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler3) [341fa7e2] Attempting to remove unneeded template/vm OVFs in Data Center 'EECS-Primary' 2017-10-01 10:20:24,567-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler3) [341fa7e2] Successfully removed unneeded template/vm OVFs in Data Center 'EECS-Primary' 2017-10-01 10:20:24,568-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler3) [341fa7e2] Lock freed to object 'EngineLock:{exclusiveLocks='[aca61eac-ed8a-484b-bbee-3645148f6405=OVF_UPDATE]', sharedLocks=''}' 2017-10-01 10:20:49,191-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [30d037e6] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 10:21:49,210-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [5db92ab9] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 10:22:49,235-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [150a0bdd] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 10:23:49,255-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [5290c29d] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 10:24:49,274-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [369dc155] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 10:25:49,296-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [ae2140b] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 10:26:49,317-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [1464c2e2] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 10:27:49,341-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [4dff7891] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 10:28:24,208-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-6-thread-19) [3836b0ae] START, GetFileStatsVDSCommand( GetFileStatsParameters:{runAsync='true', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', ignoreFailoverLimit='false'}), log id: 359e6a09 2017-10-01 10:28:24,815-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-6-thread-19) [3836b0ae] FINISH, GetFileStatsVDSCommand, return: {virtio-win-0.1.126.iso={status=0, ctime=1497024605.0, size=155856896}, server2012r2_virtio.iso={status=0, ctime=1497297085.0, size=4977590272}, en_windows_7_professional_with_sp1_x64_dvd_621750.iso={status=0, ctime=1435288543.0, size=3319478272}, virtio-win-0.1.105.iso={status=0, ctime=1435288648.0, size=56584192}}, log id: 359e6a09 2017-10-01 10:28:24,818-04 INFO [org.ovirt.engine.core.bll.storage.domain.IsoDomainListSynchronizer] (org.ovirt.thread.pool-6-thread-19) [3836b0ae] Finished automatic refresh process for 'ISO' file type with success, for storage domain id '63d680c7-4c94-4b29-a6cb-f288a128b9bc'. 2017-10-01 10:28:49,361-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [7b7c9d86] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 10:29:49,381-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [2376b849] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 10:30:49,400-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [24088a16] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 10:31:49,420-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [3a2ef303] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 10:32:49,442-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [41165faa] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 10:33:49,462-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler7) [316cd373] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 10:34:49,482-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [b262b57] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 10:35:49,501-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [f44310f] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 10:36:49,521-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [704e0b43] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 10:37:49,549-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [3fbc97cd] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 10:38:49,568-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [4e0cfc28] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 10:39:49,590-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [51aee447] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 10:40:49,610-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [5e8345f8] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 10:41:49,630-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [59814ea] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 10:42:49,654-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [23f33678] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 10:43:49,673-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [4f5b7510] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 10:44:49,694-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [498c8eed] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 10:45:49,714-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [40c4640] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 10:46:49,734-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [618cf6f0] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 10:47:49,760-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [64eb6878] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 10:48:49,779-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [f530a4f] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 10:49:49,799-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler7) [6c4bdef6] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 10:50:49,820-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [772f4221] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 10:51:49,842-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [720f1893] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 10:52:49,866-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [2be9ec2b] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 10:53:49,890-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler7) [b4888b0] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 10:54:49,909-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [726418c9] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 10:55:49,929-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [688e5bbb] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 10:56:49,948-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [3319ddfb] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 10:57:49,972-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [5a2c9477] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 10:58:49,991-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [16d5c978] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 10:59:50,011-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [76edf68f] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 11:00:50,030-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [61354042] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 11:01:50,049-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [4ba41b8d] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 11:02:50,073-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [30989194] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 11:03:50,093-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [5daeabdb] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 11:04:50,112-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [7dc94acf] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 11:05:50,132-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [52877640] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 11:06:50,152-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [52cd56ab] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 11:07:50,174-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [69ab3ce5] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 11:08:50,198-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [581d75a6] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 11:09:50,218-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler7) [4d3c34c8] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 11:10:50,241-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [22fec36c] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 11:11:50,259-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [5d435e05] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 11:12:50,283-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [5d97d255] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 11:13:50,301-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [5374461f] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 11:14:50,321-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [1c5966df] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 11:15:50,342-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [4c0a5d3e] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 11:16:50,362-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [638bf33e] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 11:17:50,388-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [4e3aede] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 11:18:50,407-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [36598b22] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 11:19:50,426-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [1e24846] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 11:20:24,568-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.OvfDataUpdater] (DefaultQuartzScheduler4) [6dedd4c3] Attempting to update VMs/Templates Ovf. 2017-10-01 11:20:24,571-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler4) [39f7505b] Before acquiring and wait lock 'EngineLock:{exclusiveLocks='[aca61eac-ed8a-484b-bbee-3645148f6405=OVF_UPDATE]', sharedLocks=''}' 2017-10-01 11:20:24,571-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler4) [39f7505b] Lock-wait acquired to object 'EngineLock:{exclusiveLocks='[aca61eac-ed8a-484b-bbee-3645148f6405=OVF_UPDATE]', sharedLocks=''}' 2017-10-01 11:20:24,572-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler4) [39f7505b] Running command: ProcessOvfUpdateForStoragePoolCommand internal: true. Entities affected : ID: aca61eac-ed8a-484b-bbee-3645148f6405 Type: StoragePool 2017-10-01 11:20:24,577-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler4) [39f7505b] Attempting to update VM OVFs in Data Center 'EECS-Primary' 2017-10-01 11:20:24,578-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler4) [39f7505b] Successfully updated VM OVFs in Data Center 'EECS-Primary' 2017-10-01 11:20:24,578-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler4) [39f7505b] Attempting to update template OVFs in Data Center 'EECS-Primary' 2017-10-01 11:20:24,578-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler4) [39f7505b] Successfully updated templates OVFs in Data Center 'EECS-Primary' 2017-10-01 11:20:24,578-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler4) [39f7505b] Attempting to remove unneeded template/vm OVFs in Data Center 'EECS-Primary' 2017-10-01 11:20:24,579-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler4) [39f7505b] Successfully removed unneeded template/vm OVFs in Data Center 'EECS-Primary' 2017-10-01 11:20:24,580-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler4) [39f7505b] Lock freed to object 'EngineLock:{exclusiveLocks='[aca61eac-ed8a-484b-bbee-3645148f6405=OVF_UPDATE]', sharedLocks=''}' 2017-10-01 11:20:50,444-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [1676d3ee] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 11:21:50,463-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [4c18319c] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 11:22:50,489-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [184a0efe] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 11:23:50,508-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [56ec0d51] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 11:24:50,531-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [10f1e42c] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 11:25:50,551-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [16abe895] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 11:26:50,570-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [52936f52] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 11:27:50,594-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [539d82f9] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 11:28:24,820-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-6-thread-32) [5788da9f] START, GetFileStatsVDSCommand( GetFileStatsParameters:{runAsync='true', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', ignoreFailoverLimit='false'}), log id: 47937c80 2017-10-01 11:28:25,888-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-6-thread-32) [5788da9f] FINISH, GetFileStatsVDSCommand, return: {virtio-win-0.1.126.iso={status=0, ctime=1497024605.0, size=155856896}, server2012r2_virtio.iso={status=0, ctime=1497297085.0, size=4977590272}, en_windows_7_professional_with_sp1_x64_dvd_621750.iso={status=0, ctime=1435288543.0, size=3319478272}, virtio-win-0.1.105.iso={status=0, ctime=1435288648.0, size=56584192}}, log id: 47937c80 2017-10-01 11:28:25,892-04 INFO [org.ovirt.engine.core.bll.storage.domain.IsoDomainListSynchronizer] (org.ovirt.thread.pool-6-thread-32) [5788da9f] Finished automatic refresh process for 'ISO' file type with success, for storage domain id '63d680c7-4c94-4b29-a6cb-f288a128b9bc'. 2017-10-01 11:28:50,616-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [474d9bbe] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 11:29:50,635-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [5f0e2354] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 11:30:50,655-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [46edcd0e] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 11:31:50,674-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [325c0ac5] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 11:32:50,698-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [6daa0187] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 11:33:50,719-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler7) [14ef23b6] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 11:34:50,742-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [22e77e33] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 11:35:50,762-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler7) [19a24450] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 11:36:50,782-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [189d5f29] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 11:37:50,809-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [a40d21] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 11:38:50,828-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler7) [529724fc] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 11:39:50,852-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [1fa73fa3] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 11:40:50,873-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [395b1a9f] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 11:41:50,893-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler7) [41fc0be2] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 11:42:50,919-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [41aa9db6] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 11:43:50,939-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [d5bb8d] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 11:44:50,958-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [2f2d8620] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 11:45:50,981-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [19291a8a] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 11:46:51,001-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [7c3b98d7] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 11:47:51,026-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [3b88dde3] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 11:48:51,043-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [65318efa] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 11:49:51,066-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [57c1b052] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 11:50:51,085-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [4c4f13de] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 11:51:51,104-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler7) [36752e88] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 11:52:51,130-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [517bddbe] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 11:53:51,152-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [358e5fee] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 11:54:51,171-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [327eddf8] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 11:55:51,188-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [1547cba9] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 11:56:51,207-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [168c19ab] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 11:57:51,233-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [18f4d0dc] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 11:58:51,252-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [3e7fcd67] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 11:59:51,271-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [4e174f3b] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 12:00:51,290-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [5ea96223] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 12:01:51,309-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [798db18] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 12:02:51,334-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [4415bd2a] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 12:03:51,353-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [799a2309] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 12:04:51,372-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [178ca472] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 12:05:51,395-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler7) [4a529d1] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 12:06:51,415-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [6cd6a603] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 12:07:51,439-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [78ffca34] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 12:08:51,458-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [5ad68f53] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 12:09:51,478-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [4eeec3eb] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 12:10:51,502-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [4055712] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 12:11:51,521-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler7) [3b3ce0f] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 12:12:51,545-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [7745e40c] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 12:13:51,565-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [4684b990] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 12:14:51,585-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [46aadfc3] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 12:15:51,605-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [f82399c] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 12:16:51,626-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [5d06a5c1] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 12:17:51,651-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [c54d717] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 12:18:51,670-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [64298ba0] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 12:19:20,457-04 INFO [org.ovirt.engine.core.bll.EngineBackupAwarenessManager] (DefaultQuartzScheduler3) [716413b] Backup check started. 2017-10-01 12:19:20,460-04 INFO [org.ovirt.engine.core.bll.EngineBackupAwarenessManager] (DefaultQuartzScheduler3) [716413b] Backup check completed. 2017-10-01 12:19:51,691-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [78532ad6] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 12:20:24,581-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.OvfDataUpdater] (DefaultQuartzScheduler7) [596a67bb] Attempting to update VMs/Templates Ovf. 2017-10-01 12:20:24,584-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler7) [495f3e8] Before acquiring and wait lock 'EngineLock:{exclusiveLocks='[aca61eac-ed8a-484b-bbee-3645148f6405=OVF_UPDATE]', sharedLocks=''}' 2017-10-01 12:20:24,584-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler7) [495f3e8] Lock-wait acquired to object 'EngineLock:{exclusiveLocks='[aca61eac-ed8a-484b-bbee-3645148f6405=OVF_UPDATE]', sharedLocks=''}' 2017-10-01 12:20:24,585-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler7) [495f3e8] Running command: ProcessOvfUpdateForStoragePoolCommand internal: true. Entities affected : ID: aca61eac-ed8a-484b-bbee-3645148f6405 Type: StoragePool 2017-10-01 12:20:24,590-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler7) [495f3e8] Attempting to update VM OVFs in Data Center 'EECS-Primary' 2017-10-01 12:20:24,594-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler7) [495f3e8] Successfully updated VM OVFs in Data Center 'EECS-Primary' 2017-10-01 12:20:24,594-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler7) [495f3e8] Attempting to update template OVFs in Data Center 'EECS-Primary' 2017-10-01 12:20:24,595-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler7) [495f3e8] Successfully updated templates OVFs in Data Center 'EECS-Primary' 2017-10-01 12:20:24,595-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler7) [495f3e8] Attempting to remove unneeded template/vm OVFs in Data Center 'EECS-Primary' 2017-10-01 12:20:24,596-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler7) [495f3e8] Successfully removed unneeded template/vm OVFs in Data Center 'EECS-Primary' 2017-10-01 12:20:24,597-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler7) [495f3e8] Lock freed to object 'EngineLock:{exclusiveLocks='[aca61eac-ed8a-484b-bbee-3645148f6405=OVF_UPDATE]', sharedLocks=''}' 2017-10-01 12:20:51,711-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [5a0e826] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 12:21:51,731-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [412e1504] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 12:22:51,755-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [27d6caf9] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 12:23:51,774-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler7) [1711aea8] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 12:24:51,793-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [5be356dc] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 12:25:51,812-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [2f6ffb2f] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 12:26:51,832-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [394260c2] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 12:27:51,859-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler7) [144af577] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 12:28:25,894-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-6-thread-31) [62fc2b6] START, GetFileStatsVDSCommand( GetFileStatsParameters:{runAsync='true', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', ignoreFailoverLimit='false'}), log id: 5288b7eb 2017-10-01 12:28:26,907-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-6-thread-31) [62fc2b6] FINISH, GetFileStatsVDSCommand, return: {virtio-win-0.1.126.iso={status=0, ctime=1497024605.0, size=155856896}, server2012r2_virtio.iso={status=0, ctime=1497297085.0, size=4977590272}, en_windows_7_professional_with_sp1_x64_dvd_621750.iso={status=0, ctime=1435288543.0, size=3319478272}, virtio-win-0.1.105.iso={status=0, ctime=1435288648.0, size=56584192}}, log id: 5288b7eb 2017-10-01 12:28:26,910-04 INFO [org.ovirt.engine.core.bll.storage.domain.IsoDomainListSynchronizer] (org.ovirt.thread.pool-6-thread-31) [62fc2b6] Finished automatic refresh process for 'ISO' file type with success, for storage domain id '63d680c7-4c94-4b29-a6cb-f288a128b9bc'. 2017-10-01 12:28:51,878-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [70b31d24] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 12:29:51,897-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [46de30b0] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 12:30:51,917-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [1dd85da1] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 12:31:51,936-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [3fa705a8] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 12:32:51,960-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [1b586bfb] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 12:33:51,980-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [42c5db9b] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 12:34:21,828-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (pool-7-thread-4) [46643b9] Connected to host virt3-mgmt with SSH key fingerprint: SHA256:uEITFypKN2NKmVr6oXEn4HzdX/nPLWoF7hCIM6wZWn4 2017-10-01 12:34:21,829-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (pool-7-thread-3) [4079836f] Connected to host virt2-mgmt with SSH key fingerprint: SHA256:9rTAzk8kU6Jf/8vUGLE0197GVa38smlfokpqluJtwh0 2017-10-01 12:34:21,830-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (pool-7-thread-2) [4079836f] Connected to host virt1-mgmt with SSH key fingerprint: SHA256:UXnYZfoFiWDh8kNO1+xfGZTr8hNs6tUUjIgHQA4wx7Y 2017-10-01 12:34:21,831-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (pool-7-thread-5) [46643b9] Connected to host virt4-mgmt with SSH key fingerprint: SHA256:sxfM1m0UOhJPws87akOxA2NL0D01f3RJm74c0r+40Ag 2017-10-01 12:34:21,907-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (pool-7-thread-4) [46643b9] Installation of virt3-mgmt. Executing command via SSH umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -d -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; tar --warning=no-timestamp -C "${MYTMP}" -x && "${MYTMP}"/ovirt-host-mgmt DIALOG/dialect=str:machine DIALOG/customization=bool:True < /var/cache/ovirt-engine/ovirt-host-deploy.tar 2017-10-01 12:34:21,907-04 INFO [org.ovirt.engine.core.utils.archivers.tar.CachedTar] (pool-7-thread-4) [46643b9] Tarball '/var/cache/ovirt-engine/ovirt-host-deploy.tar' refresh 2017-10-01 12:34:21,908-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (pool-7-thread-3) [4079836f] Installation of virt2-mgmt. Executing command via SSH umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -d -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; tar --warning=no-timestamp -C "${MYTMP}" -x && "${MYTMP}"/ovirt-host-mgmt DIALOG/dialect=str:machine DIALOG/customization=bool:True < /var/cache/ovirt-engine/ovirt-host-deploy.tar 2017-10-01 12:34:21,908-04 INFO [org.ovirt.engine.core.uutils.ssh.SSHDialog] (pool-7-thread-3) [4079836f] SSH execute 'root@virt2-mgmt' 'umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -d -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; tar --warning=no-timestamp -C "${MYTMP}" -x && "${MYTMP}"/ovirt-host-mgmt DIALOG/dialect=str:machine DIALOG/customization=bool:True' 2017-10-01 12:34:21,910-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (pool-7-thread-2) [4079836f] Installation of virt1-mgmt. Executing command via SSH umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -d -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; tar --warning=no-timestamp -C "${MYTMP}" -x && "${MYTMP}"/ovirt-host-mgmt DIALOG/dialect=str:machine DIALOG/customization=bool:True < /var/cache/ovirt-engine/ovirt-host-deploy.tar 2017-10-01 12:34:21,910-04 INFO [org.ovirt.engine.core.uutils.ssh.SSHDialog] (pool-7-thread-2) [4079836f] SSH execute 'root@virt1-mgmt' 'umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -d -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; tar --warning=no-timestamp -C "${MYTMP}" -x && "${MYTMP}"/ovirt-host-mgmt DIALOG/dialect=str:machine DIALOG/customization=bool:True' 2017-10-01 12:34:21,911-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (pool-7-thread-5) [46643b9] Installation of virt4-mgmt. Executing command via SSH umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -d -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; tar --warning=no-timestamp -C "${MYTMP}" -x && "${MYTMP}"/ovirt-host-mgmt DIALOG/dialect=str:machine DIALOG/customization=bool:True < /var/cache/ovirt-engine/ovirt-host-deploy.tar 2017-10-01 12:34:21,911-04 INFO [org.ovirt.engine.core.uutils.ssh.SSHDialog] (pool-7-thread-5) [46643b9] SSH execute 'root@virt4-mgmt' 'umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -d -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; tar --warning=no-timestamp -C "${MYTMP}" -x && "${MYTMP}"/ovirt-host-mgmt DIALOG/dialect=str:machine DIALOG/customization=bool:True' 2017-10-01 12:34:21,926-04 INFO [org.ovirt.engine.core.uutils.ssh.SSHDialog] (pool-7-thread-4) [46643b9] SSH execute 'root@virt3-mgmt' 'umask 0077; MYTMP="$(TMPDIR="${OVIRT_TMPDIR}" mktemp -d -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; tar --warning=no-timestamp -C "${MYTMP}" -x && "${MYTMP}"/ovirt-host-mgmt DIALOG/dialect=str:machine DIALOG/customization=bool:True' 2017-10-01 12:34:22,533-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [4079836f] Stage: Initializing 2017-10-01 12:34:22,538-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [46643b9] Stage: Initializing 2017-10-01 12:34:22,540-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [4079836f] Stage: Environment setup 2017-10-01 12:34:22,546-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [46643b9] Stage: Environment setup 2017-10-01 12:34:22,552-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [4079836f] Stage: Environment packages setup 2017-10-01 12:34:22,557-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [4079836f] Stage: Initializing 2017-10-01 12:34:22,559-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [46643b9] Stage: Environment packages setup 2017-10-01 12:34:22,565-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [4079836f] Stage: Environment setup 2017-10-01 12:34:22,578-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [4079836f] Stage: Environment packages setup 2017-10-01 12:34:22,595-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [46643b9] Stage: Initializing 2017-10-01 12:34:22,600-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [46643b9] Stage: Environment setup 2017-10-01 12:34:22,609-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [46643b9] Stage: Environment packages setup 2017-10-01 12:34:26,178-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [4079836f] Stage: Programs detection 2017-10-01 12:34:26,199-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [4079836f] Stage: Environment customization 2017-10-01 12:34:26,201-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [46643b9] Stage: Programs detection 2017-10-01 12:34:26,219-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [4079836f] Stage: Programs detection 2017-10-01 12:34:26,220-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [46643b9] Stage: Environment customization 2017-10-01 12:34:26,221-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [4079836f] Logs at host located at: '/tmp/ovirt-host-mgmt-20171001123422-xzf9ib.log' 2017-10-01 12:34:26,234-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [4079836f] Stage: Setup validation 2017-10-01 12:34:26,238-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [4079836f] Stage: Environment customization 2017-10-01 12:34:26,243-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [46643b9] Logs at host located at: '/tmp/ovirt-host-mgmt-20171001123422-zzfb6n.log' 2017-10-01 12:34:26,249-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [4079836f] Stage: Transaction setup 2017-10-01 12:34:26,254-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [46643b9] Stage: Setup validation 2017-10-01 12:34:26,260-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [4079836f] Logs at host located at: '/tmp/ovirt-host-mgmt-20171001123422-bdgrxs.log' 2017-10-01 12:34:26,269-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [46643b9] Stage: Transaction setup 2017-10-01 12:34:26,284-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [46643b9] Stage: Programs detection 2017-10-01 12:34:26,296-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [4079836f] Stage: Misc configuration 2017-10-01 12:34:26,297-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [4079836f] Stage: Package installation 2017-10-01 12:34:26,301-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [46643b9] Stage: Environment customization 2017-10-01 12:34:26,310-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [4079836f] Stage: Setup validation 2017-10-01 12:34:26,318-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [46643b9] Stage: Misc configuration 2017-10-01 12:34:26,319-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [46643b9] Stage: Package installation 2017-10-01 12:34:26,320-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [46643b9] Logs at host located at: '/tmp/ovirt-host-mgmt-20171001123422-ydb1jq.log' 2017-10-01 12:34:26,330-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [46643b9] Stage: Setup validation 2017-10-01 12:34:26,338-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [4079836f] Stage: Transaction setup 2017-10-01 12:34:26,344-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [46643b9] Stage: Transaction setup 2017-10-01 12:34:26,382-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [46643b9] Stage: Misc configuration 2017-10-01 12:34:26,383-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [46643b9] Stage: Package installation 2017-10-01 12:34:26,385-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [4079836f] Stage: Misc configuration 2017-10-01 12:34:26,386-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [4079836f] Stage: Package installation 2017-10-01 12:34:34,111-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [46643b9] Stage: Misc configuration 2017-10-01 12:34:34,114-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [46643b9] Stage: Transaction commit 2017-10-01 12:34:34,151-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [46643b9] Stage: Closing up 2017-10-01 12:34:34,153-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [46643b9] Stage: Pre-termination 2017-10-01 12:34:34,165-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [46643b9] Retrieving installation logs to: '/var/log/ovirt-engine/host-deploy/ovirt-host-mgmt-20171001123434-virt4-mgmt-46643b9.log' 2017-10-01 12:34:34,228-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [46643b9] Stage: Termination 2017-10-01 12:34:34,291-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-7-thread-5) [46643b9] EVENT_ID: HOST_AVAILABLE_UPDATES_FINISHED(885), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Check for available updates on host virt4 was completed successfully with message 'no updates found.'. 2017-10-01 12:34:35,752-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [4079836f] Stage: Misc configuration 2017-10-01 12:34:35,757-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [4079836f] Stage: Transaction commit 2017-10-01 12:34:35,805-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [4079836f] Stage: Closing up 2017-10-01 12:34:35,808-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [4079836f] Stage: Pre-termination 2017-10-01 12:34:35,822-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [4079836f] Retrieving installation logs to: '/var/log/ovirt-engine/host-deploy/ovirt-host-mgmt-20171001123435-virt1-mgmt-4079836f.log' 2017-10-01 12:34:35,914-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [4079836f] Stage: Termination 2017-10-01 12:34:35,971-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-7-thread-2) [4079836f] EVENT_ID: HOST_AVAILABLE_UPDATES_FINISHED(885), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Check for available updates on host virt1 was completed successfully with message 'no updates found.'. 2017-10-01 12:34:36,041-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [4079836f] Stage: Misc configuration 2017-10-01 12:34:36,046-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [4079836f] Stage: Transaction commit 2017-10-01 12:34:36,094-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [4079836f] Stage: Closing up 2017-10-01 12:34:36,097-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [4079836f] Stage: Pre-termination 2017-10-01 12:34:36,111-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [4079836f] Retrieving installation logs to: '/var/log/ovirt-engine/host-deploy/ovirt-host-mgmt-20171001123436-virt2-mgmt-4079836f.log' 2017-10-01 12:34:36,196-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [4079836f] Stage: Termination 2017-10-01 12:34:36,250-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-7-thread-3) [4079836f] EVENT_ID: HOST_AVAILABLE_UPDATES_FINISHED(885), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Check for available updates on host virt2 was completed successfully with message 'no updates found.'. 2017-10-01 12:34:36,872-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [46643b9] Stage: Misc configuration 2017-10-01 12:34:36,877-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [46643b9] Stage: Transaction commit 2017-10-01 12:34:36,923-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [46643b9] Stage: Closing up 2017-10-01 12:34:36,926-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [46643b9] Stage: Pre-termination 2017-10-01 12:34:36,939-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [46643b9] Retrieving installation logs to: '/var/log/ovirt-engine/host-deploy/ovirt-host-mgmt-20171001123436-virt3-mgmt-46643b9.log' 2017-10-01 12:34:37,023-04 INFO [org.ovirt.engine.core.bll.hostdeploy.VdsDeployBase] (VdsDeploy) [46643b9] Stage: Termination 2017-10-01 12:34:37,078-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-7-thread-4) [46643b9] EVENT_ID: HOST_AVAILABLE_UPDATES_FINISHED(885), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Check for available updates on host virt3 was completed successfully with message 'no updates found.'. 2017-10-01 12:34:51,999-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [3eeb2935] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 12:35:52,019-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [5ed214cb] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 12:36:52,038-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [7762c615] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 12:37:52,062-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [2c7bcd13] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 12:38:52,079-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [2823ab54] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 12:39:52,102-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [6f1fdcb2] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 12:40:52,122-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler7) [3a5078] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 12:41:52,145-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [6df4f89f] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 12:42:52,169-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [7587593e] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 12:43:52,188-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [2f9c4df6] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 12:44:52,210-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [52a727e0] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 12:45:52,230-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [1e5b6745] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 12:46:52,249-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [45c93547] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 12:47:52,274-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [2fb2513b] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 12:48:52,294-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [78cf457c] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 12:49:52,317-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [253fd599] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 12:50:52,336-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [51e06d37] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 12:51:52,356-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [b0bf656] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 12:52:52,381-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [4f4f883a] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 12:53:52,405-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [59d2d274] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 12:54:52,425-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [2574f4f7] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 12:55:52,442-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [4fc2f5ce] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 12:56:52,461-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [26ccf221] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 12:57:52,488-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [72f91c5c] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 12:58:52,509-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [224178e9] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 12:59:52,528-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [2c346b9] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 13:00:52,547-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [119a1691] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 13:01:52,568-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [3458621] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 13:02:52,593-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [1884652b] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 13:03:52,612-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [5b165111] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 13:04:52,630-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [6b401667] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 13:05:52,650-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [667c50d1] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 13:06:52,670-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [2c506f1d] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 13:07:52,691-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [14f8be7a] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 13:08:52,711-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [7901d6ed] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 13:09:52,730-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [1013aa85] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 13:10:52,749-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [21a6f60] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 13:11:52,768-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [7dfff4ae] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 13:12:52,798-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [4c4a7acf] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 13:13:52,817-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [7f5a9abc] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 13:14:52,838-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [72347a2c] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 13:15:52,855-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [3afc1362] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 13:16:52,874-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [29b49c07] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 13:17:52,899-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [2cfbe21] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 13:18:52,918-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [79f0f59f] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 13:19:52,938-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [7802ae63] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 13:20:24,598-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.OvfDataUpdater] (DefaultQuartzScheduler8) [74fad801] Attempting to update VMs/Templates Ovf. 2017-10-01 13:20:24,601-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler8) [3f880b2b] Before acquiring and wait lock 'EngineLock:{exclusiveLocks='[aca61eac-ed8a-484b-bbee-3645148f6405=OVF_UPDATE]', sharedLocks=''}' 2017-10-01 13:20:24,601-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler8) [3f880b2b] Lock-wait acquired to object 'EngineLock:{exclusiveLocks='[aca61eac-ed8a-484b-bbee-3645148f6405=OVF_UPDATE]', sharedLocks=''}' 2017-10-01 13:20:24,602-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler8) [3f880b2b] Running command: ProcessOvfUpdateForStoragePoolCommand internal: true. Entities affected : ID: aca61eac-ed8a-484b-bbee-3645148f6405 Type: StoragePool 2017-10-01 13:20:24,607-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler8) [3f880b2b] Attempting to update VM OVFs in Data Center 'EECS-Primary' 2017-10-01 13:20:24,608-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler8) [3f880b2b] Successfully updated VM OVFs in Data Center 'EECS-Primary' 2017-10-01 13:20:24,608-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler8) [3f880b2b] Attempting to update template OVFs in Data Center 'EECS-Primary' 2017-10-01 13:20:24,608-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler8) [3f880b2b] Successfully updated templates OVFs in Data Center 'EECS-Primary' 2017-10-01 13:20:24,608-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler8) [3f880b2b] Attempting to remove unneeded template/vm OVFs in Data Center 'EECS-Primary' 2017-10-01 13:20:24,609-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler8) [3f880b2b] Successfully removed unneeded template/vm OVFs in Data Center 'EECS-Primary' 2017-10-01 13:20:24,610-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler8) [3f880b2b] Lock freed to object 'EngineLock:{exclusiveLocks='[aca61eac-ed8a-484b-bbee-3645148f6405=OVF_UPDATE]', sharedLocks=''}' 2017-10-01 13:20:52,956-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [183e32b7] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 13:21:52,976-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler7) [6d4bb19] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 13:22:53,000-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [35d1a405] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 13:23:53,020-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [23eafe80] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 13:24:53,040-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [728e174b] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 13:25:53,058-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [3357c01c] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 13:26:53,077-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [4a7cef59] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 13:27:53,101-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [4fc6d084] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 13:28:26,914-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-6-thread-9) [fe540ab] START, GetFileStatsVDSCommand( GetFileStatsParameters:{runAsync='true', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', ignoreFailoverLimit='false'}), log id: 50198e70 2017-10-01 13:28:27,926-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-6-thread-9) [fe540ab] FINISH, GetFileStatsVDSCommand, return: {virtio-win-0.1.126.iso={status=0, ctime=1497024605.0, size=155856896}, server2012r2_virtio.iso={status=0, ctime=1497297085.0, size=4977590272}, en_windows_7_professional_with_sp1_x64_dvd_621750.iso={status=0, ctime=1435288543.0, size=3319478272}, virtio-win-0.1.105.iso={status=0, ctime=1435288648.0, size=56584192}}, log id: 50198e70 2017-10-01 13:28:27,930-04 INFO [org.ovirt.engine.core.bll.storage.domain.IsoDomainListSynchronizer] (org.ovirt.thread.pool-6-thread-9) [fe540ab] Finished automatic refresh process for 'ISO' file type with success, for storage domain id '63d680c7-4c94-4b29-a6cb-f288a128b9bc'. 2017-10-01 13:28:53,118-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [52cd846] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 13:29:53,141-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [12385329] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 13:30:53,160-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [39ce6267] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 13:31:53,180-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [7a5e9dec] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 13:32:53,205-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [22559594] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 13:33:53,224-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [5914c189] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 13:34:53,246-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [57808d8f] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 13:35:53,265-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [b912bdc] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 13:36:53,283-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [a5eaa39] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 13:37:53,307-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [1400d1a] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 13:38:53,326-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [73af8fc] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 13:39:53,347-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [717bef62] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 13:40:53,366-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [3f2f530a] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 13:41:53,385-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [4c044739] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 13:42:53,409-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [37754cfd] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 13:43:53,432-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [54fa115e] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 13:44:53,452-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [1abad85f] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 13:45:53,471-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [58677401] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 13:46:53,490-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [220c5006] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 13:47:53,512-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [6188aa1f] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 13:48:53,532-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [13662791] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 13:49:53,553-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler7) [2bdb2159] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 13:50:53,569-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler7) [3c1351cc] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 13:51:53,588-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [775b9453] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 13:52:53,612-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [22e67c93] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 13:53:53,631-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [2431a868] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 13:54:53,651-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [5a54e1e3] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 13:55:53,667-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [7038e554] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 13:56:53,688-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [4b4044e4] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 13:57:53,712-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [27aa6222] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 13:58:53,731-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler7) [2a6b4d5b] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 13:59:53,750-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [3d5f1fea] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 14:00:53,768-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [2a8703a9] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 14:01:53,789-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [6706c49e] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 14:02:53,814-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [3e0299ad] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 14:03:53,834-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [8f5ce9e] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 14:04:53,853-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler7) [5159617b] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 14:05:53,878-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [6ce32441] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 14:06:53,897-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [eab841a] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 14:07:53,921-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [7536846d] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 14:08:53,941-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [7d45bac] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 14:09:53,960-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [87822e9] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 14:10:53,980-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [28998a3f] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 14:11:53,997-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [331dceef] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 14:12:54,021-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [4585c321] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 14:13:54,040-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [42db0ddd] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 14:14:54,059-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [20d66e47] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 14:15:54,079-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [61203417] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 14:16:54,098-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [63e4cd30] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 14:17:54,125-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [5f71bfb2] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 14:18:54,145-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [50d4003a] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 14:19:54,164-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [37ba0b3d] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 14:20:24,610-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.OvfDataUpdater] (DefaultQuartzScheduler6) [2b4e7ccb] Attempting to update VMs/Templates Ovf. 2017-10-01 14:20:24,613-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [7d9028c1] Before acquiring and wait lock 'EngineLock:{exclusiveLocks='[aca61eac-ed8a-484b-bbee-3645148f6405=OVF_UPDATE]', sharedLocks=''}' 2017-10-01 14:20:24,613-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [7d9028c1] Lock-wait acquired to object 'EngineLock:{exclusiveLocks='[aca61eac-ed8a-484b-bbee-3645148f6405=OVF_UPDATE]', sharedLocks=''}' 2017-10-01 14:20:24,614-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [7d9028c1] Running command: ProcessOvfUpdateForStoragePoolCommand internal: true. Entities affected : ID: aca61eac-ed8a-484b-bbee-3645148f6405 Type: StoragePool 2017-10-01 14:20:24,681-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [7d9028c1] Attempting to update VM OVFs in Data Center 'EECS-Primary' 2017-10-01 14:20:24,681-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [7d9028c1] Successfully updated VM OVFs in Data Center 'EECS-Primary' 2017-10-01 14:20:24,682-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [7d9028c1] Attempting to update template OVFs in Data Center 'EECS-Primary' 2017-10-01 14:20:24,682-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [7d9028c1] Successfully updated templates OVFs in Data Center 'EECS-Primary' 2017-10-01 14:20:24,682-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [7d9028c1] Attempting to remove unneeded template/vm OVFs in Data Center 'EECS-Primary' 2017-10-01 14:20:24,683-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [7d9028c1] Successfully removed unneeded template/vm OVFs in Data Center 'EECS-Primary' 2017-10-01 14:20:24,684-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler6) [7d9028c1] Lock freed to object 'EngineLock:{exclusiveLocks='[aca61eac-ed8a-484b-bbee-3645148f6405=OVF_UPDATE]', sharedLocks=''}' 2017-10-01 14:20:54,183-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [78da504d] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 14:21:54,201-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [13891de9] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 14:22:54,225-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [e3117d6] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 14:23:54,244-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [5c338ab2] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 14:24:54,263-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler7) [5b6fb908] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 14:25:54,282-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [7ab982e6] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 14:26:54,299-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [77a90b07] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 14:27:54,324-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [70952f4b] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 14:28:03,534-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler5) [7c1f8427] VM 'c64d80f9-a3bd-44f3-89c4-f1b9b509cdde'(red) moved from 'Up' --> 'NotResponding' 2017-10-01 14:28:03,607-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [7c1f8427] EVENT_ID: VM_NOT_RESPONDING(126), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM red is not responding. 2017-10-01 14:28:27,933-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-6-thread-39) [59ff0296] START, GetFileStatsVDSCommand( GetFileStatsParameters:{runAsync='true', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', ignoreFailoverLimit='false'}), log id: 4e67278d 2017-10-01 14:28:28,945-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-6-thread-39) [59ff0296] FINISH, GetFileStatsVDSCommand, return: {virtio-win-0.1.126.iso={status=0, ctime=1497024605.0, size=155856896}, server2012r2_virtio.iso={status=0, ctime=1497297085.0, size=4977590272}, en_windows_7_professional_with_sp1_x64_dvd_621750.iso={status=0, ctime=1435288543.0, size=3319478272}, virtio-win-0.1.105.iso={status=0, ctime=1435288648.0, size=56584192}}, log id: 4e67278d 2017-10-01 14:28:28,949-04 INFO [org.ovirt.engine.core.bll.storage.domain.IsoDomainListSynchronizer] (org.ovirt.thread.pool-6-thread-39) [59ff0296] Finished automatic refresh process for 'ISO' file type with success, for storage domain id '63d680c7-4c94-4b29-a6cb-f288a128b9bc'. 2017-10-01 14:28:54,344-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [29751ac7] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 14:29:54,363-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [a426bad] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 14:30:54,382-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler7) [2324ec5a] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 14:31:54,405-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [21991f69] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 14:32:54,430-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler7) [46cc7f42] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 14:33:54,449-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [3c8e999] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 14:34:54,468-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [2c4f012f] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 14:35:54,484-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [713d31e1] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 14:36:54,503-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [35136533] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 14:37:54,526-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler7) [4f046658] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 14:38:54,545-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [196992d7] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 14:39:54,564-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [599e4ca4] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 14:40:54,581-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [211c207] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 14:41:54,601-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [221c47a1] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 14:42:54,623-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [69af649f] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 14:43:54,640-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [6f0580ea] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 14:44:54,660-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [551e7f00] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 14:45:54,677-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [4c3623f6] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 14:46:54,696-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [64e5fe4a] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 14:47:50,419-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler9) [2350534e] VM 'c64d80f9-a3bd-44f3-89c4-f1b9b509cdde'(red) moved from 'NotResponding' --> 'Paused' 2017-10-01 14:47:50,456-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [2350534e] EVENT_ID: VM_PAUSED(1,025), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM red has been paused. 2017-10-01 14:47:50,478-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [2350534e] EVENT_ID: VM_PAUSED_EIO(145), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM red has been paused due to storage I/O problem. 2017-10-01 14:47:54,720-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [7f0ea3e0] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 14:48:54,742-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [76522b14] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 14:49:06,098-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler10) [4facd1b5] VM 'c64d80f9-a3bd-44f3-89c4-f1b9b509cdde'(red) moved from 'Paused' --> 'NotResponding' 2017-10-01 14:49:06,118-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler9) [3401178c] VM 'ae56c31c-71f7-433a-b198-a1c2c846325e'(forum) moved from 'Up' --> 'NotResponding' 2017-10-01 14:49:06,129-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler1) [16726133] VM '07331c78-6e09-41b4-9b29-edda32f46351'(samba) moved from 'Up' --> 'NotResponding' 2017-10-01 14:49:06,152-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [3401178c] EVENT_ID: VM_NOT_RESPONDING(126), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM forum is not responding. 2017-10-01 14:49:06,153-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler9) [3401178c] VM 'a82b4cc3-6ef6-4a8e-97ef-1d4ea1b7e07c'(indigo) moved from 'Up' --> 'NotResponding' 2017-10-01 14:49:06,183-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [3401178c] EVENT_ID: VM_NOT_RESPONDING(126), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM indigo is not responding. 2017-10-01 14:49:06,183-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler9) [3401178c] VM 'a8d4e149-55ec-4961-a414-d7d1e1b05a59'(fix) moved from 'Up' --> 'NotResponding' 2017-10-01 14:49:06,185-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [4facd1b5] EVENT_ID: VM_NOT_RESPONDING(126), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM red is not responding. 2017-10-01 14:49:06,185-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler10) [4facd1b5] VM '1734fed5-fc97-4714-bb63-55e8f4eb6d34'(wiki) moved from 'Up' --> 'NotResponding' 2017-10-01 14:49:06,203-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [16726133] EVENT_ID: VM_NOT_RESPONDING(126), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM samba is not responding. 2017-10-01 14:49:06,203-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler1) [16726133] VM 'dc66c051-24b7-4cb8-99eb-db398d396ef1'(samba1) moved from 'Up' --> 'NotResponding' 2017-10-01 14:49:06,217-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [3401178c] EVENT_ID: VM_NOT_RESPONDING(126), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM fix is not responding. 2017-10-01 14:49:06,217-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler9) [3401178c] VM '3bb83ad5-dc00-445d-8708-2db18c53e0e6'(www-vhost) moved from 'Up' --> 'NotResponding' 2017-10-01 14:49:06,217-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [4facd1b5] EVENT_ID: VM_NOT_RESPONDING(126), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM wiki is not responding. 2017-10-01 14:49:06,218-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler10) [4facd1b5] VM '1c7e966f-29e8-4788-b64c-86e52f415061'(udb) moved from 'Up' --> 'NotResponding' 2017-10-01 14:49:06,230-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [16726133] EVENT_ID: VM_NOT_RESPONDING(126), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM samba1 is not responding. 2017-10-01 14:49:06,230-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler1) [16726133] VM 'bcd5776e-978b-4723-933d-70a80322a796'(cape) moved from 'Up' --> 'NotResponding' 2017-10-01 14:49:06,239-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [3401178c] EVENT_ID: VM_NOT_RESPONDING(126), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM www-vhost is not responding. 2017-10-01 14:49:06,239-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler9) [3401178c] VM '4ba80442-b361-4026-ae8c-895fd28042b0'(dist) moved from 'Up' --> 'NotResponding' 2017-10-01 14:49:06,240-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [4facd1b5] EVENT_ID: VM_NOT_RESPONDING(126), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM udb is not responding. 2017-10-01 14:49:06,241-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler10) [4facd1b5] VM 'cc1dbe93-bb80-451e-8b3b-c9cc5e90432f'(license) moved from 'Up' --> 'NotResponding' 2017-10-01 14:49:06,255-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [16726133] EVENT_ID: VM_NOT_RESPONDING(126), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM cape is not responding. 2017-10-01 14:49:06,255-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler1) [16726133] VM '3e5a5728-8940-4fe7-b4af-11fb8e12f116'(www) moved from 'Up' --> 'NotResponding' 2017-10-01 14:49:06,267-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [3401178c] EVENT_ID: VM_NOT_RESPONDING(126), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM dist is not responding. 2017-10-01 14:49:06,267-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [4facd1b5] EVENT_ID: VM_NOT_RESPONDING(126), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM license is not responding. 2017-10-01 14:49:06,268-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler10) [4facd1b5] VM '9ea2821e-557f-45cb-9872-1e6c5aeaf246'(rs) moved from 'Up' --> 'NotResponding' 2017-10-01 14:49:06,277-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [16726133] EVENT_ID: VM_NOT_RESPONDING(126), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM www is not responding. 2017-10-01 14:49:06,277-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler1) [16726133] VM 'f87ec08d-0fdf-42ba-8aac-c66c404d7aae'(webapp) moved from 'Up' --> 'NotResponding' 2017-10-01 14:49:06,291-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [4facd1b5] EVENT_ID: VM_NOT_RESPONDING(126), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM rs is not responding. 2017-10-01 14:49:06,292-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler10) [4facd1b5] VM '64734fca-4d6c-443c-82d1-abf7cf106601'(webdav) moved from 'Up' --> 'NotResponding' 2017-10-01 14:49:06,298-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [16726133] EVENT_ID: VM_NOT_RESPONDING(126), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM webapp is not responding. 2017-10-01 14:49:06,299-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler1) [16726133] VM 'd66235b4-7c8d-4da2-a572-82ecc9c53a59'(www-las) moved from 'Up' --> 'NotResponding' 2017-10-01 14:49:06,322-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [16726133] EVENT_ID: VM_NOT_RESPONDING(126), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM www-las is not responding. 2017-10-01 14:49:06,323-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler1) [16726133] VM '5885c43a-b133-45dc-b8d5-1aa2a4a19579'(prs) moved from 'Up' --> 'NotResponding' 2017-10-01 14:49:06,347-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [16726133] EVENT_ID: VM_NOT_RESPONDING(126), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM prs is not responding. 2017-10-01 14:49:06,347-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler1) [16726133] VM 'ae65a104-ed3f-40e0-ba92-5708cf338918'(vpn2) moved from 'Up' --> 'NotResponding' 2017-10-01 14:49:06,369-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [16726133] EVENT_ID: VM_NOT_RESPONDING(126), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM vpn2 is not responding. 2017-10-01 14:49:06,370-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler1) [16726133] VM 'd56363ad-9cb1-4a70-a33a-ce439b044d7d'(svn) moved from 'Up' --> 'NotResponding' 2017-10-01 14:49:06,388-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [4facd1b5] EVENT_ID: VM_NOT_RESPONDING(126), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM webdav is not responding. 2017-10-01 14:49:06,391-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [16726133] EVENT_ID: VM_NOT_RESPONDING(126), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM svn is not responding. 2017-10-01 14:49:08,464-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler6) [6fdf7c96] VM 'db3e08fb-c111-442d-aeec-1507da30f486'(print) moved from 'Up' --> 'NotResponding' 2017-10-01 14:49:08,490-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [6fdf7c96] EVENT_ID: VM_NOT_RESPONDING(126), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM print is not responding. 2017-10-01 14:49:08,490-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler6) [6fdf7c96] VM '911d6883-607c-464e-b46b-8e6141bdd2de'(hdb) moved from 'Up' --> 'NotResponding' 2017-10-01 14:49:08,517-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [6fdf7c96] EVENT_ID: VM_NOT_RESPONDING(126), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM hdb is not responding. 2017-10-01 14:49:08,518-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler6) [6fdf7c96] VM 'ddf37076-e795-4c87-a4fb-9efcaae76d72'(dc1) moved from 'Up' --> 'NotResponding' 2017-10-01 14:49:08,539-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [6fdf7c96] EVENT_ID: VM_NOT_RESPONDING(126), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM dc1 is not responding. 2017-10-01 14:49:08,539-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler6) [6fdf7c96] VM '899e3a1c-dcc4-4426-9dd4-0f1e8b94a2b8'(ipa) moved from 'Up' --> 'NotResponding' 2017-10-01 14:49:08,560-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [6fdf7c96] EVENT_ID: VM_NOT_RESPONDING(126), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM ipa is not responding. 2017-10-01 14:49:08,561-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler6) [6fdf7c96] VM 'bfe9a524-2238-4bab-a91f-5dc6ed270896'(pcmode) moved from 'Up' --> 'NotResponding' 2017-10-01 14:49:08,582-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [6fdf7c96] EVENT_ID: VM_NOT_RESPONDING(126), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM pcmode is not responding. 2017-10-01 14:49:08,582-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler6) [6fdf7c96] VM '1325cebf-834e-4571-a3e1-82beb0021ffe'(dns1) moved from 'Up' --> 'NotResponding' 2017-10-01 14:49:08,603-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [6fdf7c96] EVENT_ID: VM_NOT_RESPONDING(126), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM dns1 is not responding. 2017-10-01 14:49:08,603-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler6) [6fdf7c96] VM '77766cb4-0625-4432-a16f-def5e702102a'(labtest1) moved from 'Up' --> 'NotResponding' 2017-10-01 14:49:08,624-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [6fdf7c96] EVENT_ID: VM_NOT_RESPONDING(126), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM labtest1 is not responding. 2017-10-01 14:49:08,624-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler6) [6fdf7c96] VM '8391f9a7-7fe7-4025-8fba-6273ae587a52'(archive) moved from 'Up' --> 'NotResponding' 2017-10-01 14:49:08,645-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [6fdf7c96] EVENT_ID: VM_NOT_RESPONDING(126), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM archive is not responding. 2017-10-01 14:49:08,645-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler6) [6fdf7c96] VM 'c8682859-9e55-460a-9ef0-d03e4a85f50e'(dhcp1) moved from 'Up' --> 'NotResponding' 2017-10-01 14:49:08,666-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [6fdf7c96] EVENT_ID: VM_NOT_RESPONDING(126), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM dhcp1 is not responding. 2017-10-01 14:49:08,666-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler6) [6fdf7c96] VM 'aa487207-7ff4-465a-9d9b-2a103d50dc77'(ftp1) moved from 'Up' --> 'NotResponding' 2017-10-01 14:49:08,687-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [6fdf7c96] EVENT_ID: VM_NOT_RESPONDING(126), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM ftp1 is not responding. 2017-10-01 14:49:08,687-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler6) [6fdf7c96] VM '59287796-c634-4f3c-94b1-2c9699986637'(vpn1) moved from 'Up' --> 'NotResponding' 2017-10-01 14:49:08,708-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [6fdf7c96] EVENT_ID: VM_NOT_RESPONDING(126), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM vpn1 is not responding. 2017-10-01 14:49:16,496-04 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.Reactor] (SSL Stomp Reactor) [] Unable to process messages Connection reset by peer 2017-10-01 14:49:16,496-04 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.Reactor] (SSL Stomp Reactor) [] Unable to process messages Connection reset by peer 2017-10-01 14:49:21,274-04 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to virt2-mgmt/192.168.0.35 2017-10-01 14:49:21,277-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler1) [41b07d59] Command 'GetAllVmStatsVDSCommand(HostName = virt2, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e'})' execution failed: java.net.ConnectException: Connection refused 2017-10-01 14:49:21,279-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.PollVmStatsRefresher] (DefaultQuartzScheduler1) [41b07d59] Failed to fetch vms info for host 'virt2' - skipping VMs monitoring. 2017-10-01 14:49:21,279-04 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (org.ovirt.thread.pool-6-thread-20) [41b07d59] Host 'virt2' is not responding. It will stay in Connecting state for a grace period of 82 seconds and after that an attempt to fence the host will be issued. 2017-10-01 14:49:21,298-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-20) [41b07d59] EVENT_ID: VDS_HOST_NOT_RESPONDING_CONNECTING(9,008), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host virt2 is not responding. It will stay in Connecting state for a grace period of 82 seconds and after that an attempt to fence the host will be issued. 2017-10-01 14:49:22,084-04 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to virt2-mgmt/192.168.0.35 2017-10-01 14:49:22,085-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler9) [1b87e9b1] Command 'GetCapabilitiesVDSCommand(HostName = virt2, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vds='Host[virt2,a0ccfd11-3402-4eba-a119-4987561b738e]'})' execution failed: java.net.ConnectException: Connection refused 2017-10-01 14:49:22,085-04 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [1b87e9b1] Failure to refresh host 'virt2' runtime info: java.net.ConnectException: Connection refused 2017-10-01 14:49:22,085-04 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler9) [1b87e9b1] Failed to refresh VDS, network error, continuing, vds='virt2'(a0ccfd11-3402-4eba-a119-4987561b738e): java.net.ConnectException: Connection refused 2017-10-01 14:49:22,168-04 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to virt2-mgmt/192.168.0.35 2017-10-01 14:49:22,169-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler1) [26891a06] Command 'SpmStatusVDSCommand(HostName = virt2, SpmStatusVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: java.net.ConnectException: Connection refused 2017-10-01 14:49:22,179-04 INFO [org.ovirt.engine.core.bll.storage.pool.SetStoragePoolStatusCommand] (DefaultQuartzScheduler1) [19ae77cf] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: aca61eac-ed8a-484b-bbee-3645148f6405 Type: StoragePool 2017-10-01 14:49:22,180-04 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler1) [19ae77cf] Storage Pool 'aca61eac-ed8a-484b-bbee-3645148f6405' - Updating Storage Domain '63d680c7-4c94-4b29-a6cb-f288a128b9bc' status from 'Active' to 'Unknown', reason: null 2017-10-01 14:49:22,181-04 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler1) [19ae77cf] Storage Pool 'aca61eac-ed8a-484b-bbee-3645148f6405' - Updating Storage Domain 'f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2' status from 'Active' to 'Unknown', reason: null 2017-10-01 14:49:22,181-04 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler1) [19ae77cf] Storage Pool 'aca61eac-ed8a-484b-bbee-3645148f6405' - Updating Storage Domain 'c8653dba-63ec-4447-872c-0ffcc6af9c0f' status from 'Active' to 'Unknown', reason: null 2017-10-01 14:49:22,203-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [19ae77cf] EVENT_ID: SYSTEM_CHANGE_STORAGE_POOL_STATUS_PROBLEMATIC_WITH_ERROR(987), Correlation ID: 19ae77cf, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Invalid status on Data Center EECS-Primary. Setting Data Center status to Non Responsive (On host virt2, Error: Network error during communication with the Host.). 2017-10-01 14:49:22,209-04 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to virt2-mgmt/192.168.0.35 2017-10-01 14:49:22,210-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler1) [19ae77cf] IrsBroker::Failed::GetStoragePoolInfoVDS 2017-10-01 14:49:22,210-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.GetStoragePoolInfoVDSCommand] (DefaultQuartzScheduler1) [19ae77cf] Command 'GetStoragePoolInfoVDSCommand( GetStoragePoolInfoVDSCommandParameters:{runAsync='true', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', ignoreFailoverLimit='true'})' execution failed: IRSProtocolException: 2017-10-01 14:49:25,101-04 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to virt2-mgmt/192.168.0.35 2017-10-01 14:49:25,102-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler7) [5e68ffa8] Command 'GetCapabilitiesVDSCommand(HostName = virt2, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vds='Host[virt2,a0ccfd11-3402-4eba-a119-4987561b738e]'})' execution failed: java.net.ConnectException: Connection refused 2017-10-01 14:49:25,103-04 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [5e68ffa8] Failure to refresh host 'virt2' runtime info: java.net.ConnectException: Connection refused 2017-10-01 14:49:25,103-04 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler7) [5e68ffa8] Failed to refresh VDS, network error, continuing, vds='virt2'(a0ccfd11-3402-4eba-a119-4987561b738e): java.net.ConnectException: Connection refused 2017-10-01 14:49:28,117-04 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to virt2-mgmt/192.168.0.35 2017-10-01 14:49:28,118-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler9) [588fbf2b] Command 'GetCapabilitiesVDSCommand(HostName = virt2, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vds='Host[virt2,a0ccfd11-3402-4eba-a119-4987561b738e]'})' execution failed: java.net.ConnectException: Connection refused 2017-10-01 14:49:28,118-04 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [588fbf2b] Failure to refresh host 'virt2' runtime info: java.net.ConnectException: Connection refused 2017-10-01 14:49:28,118-04 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler9) [588fbf2b] Failed to refresh VDS, network error, continuing, vds='virt2'(a0ccfd11-3402-4eba-a119-4987561b738e): java.net.ConnectException: Connection refused 2017-10-01 14:49:31,133-04 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to virt2-mgmt/192.168.0.35 2017-10-01 14:49:32,257-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [26ba7645] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command GetCapabilitiesVDS failed: Recovering from crash or Initializing 2017-10-01 14:49:32,257-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler5) [26ba7645] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@285aa7b4' 2017-10-01 14:49:32,257-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler5) [26ba7645] HostName = virt2 2017-10-01 14:49:32,257-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler5) [26ba7645] Command 'GetCapabilitiesVDSCommand(HostName = virt2, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vds='Host[virt2,a0ccfd11-3402-4eba-a119-4987561b738e]'})' execution failed: Recovering from crash or Initializing 2017-10-01 14:49:32,766-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [26ba7645] EVENT_ID: VDS_INITIALIZING(514), Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSRecoveringException: Recovering from crash or Initializing 2017-10-01 14:49:32,766-04 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler5) [26ba7645] Failed to refresh VDS, continuing, vds='virt2'(a0ccfd11-3402-4eba-a119-4987561b738e): Recovering from crash or Initializing 2017-10-01 14:49:33,259-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3531151] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command SpmStatusVDS failed: Recovering from crash or Initializing 2017-10-01 14:49:33,259-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler2) [3531151] Command 'SpmStatusVDSCommand(HostName = virt2, SpmStatusVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: Recovering from crash or Initializing 2017-10-01 14:49:34,267-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3531151] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command HSMGetAllTasksStatusesVDS failed: Recovering from crash or Initializing 2017-10-01 14:49:34,267-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler2) [3531151] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand' return value ' 2017-10-01 14:49:34,267-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler2) [3531151] HostName = virt2 2017-10-01 14:49:34,267-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler2) [3531151] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = virt2, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e'})' execution failed: Recovering from crash or Initializing 2017-10-01 14:49:34,383-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [3531151] START, ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 229e55d6 2017-10-01 14:49:34,383-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [3531151] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 14:49:36,303-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [537c1554] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command GetCapabilitiesVDS failed: Recovering from crash or Initializing 2017-10-01 14:49:36,303-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler9) [537c1554] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@17d1e07e' 2017-10-01 14:49:36,303-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler9) [537c1554] HostName = virt2 2017-10-01 14:49:36,303-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler9) [537c1554] Command 'GetCapabilitiesVDSCommand(HostName = virt2, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vds='Host[virt2,a0ccfd11-3402-4eba-a119-4987561b738e]'})' execution failed: Recovering from crash or Initializing 2017-10-01 14:49:36,424-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler3) [6a80bf7c] Command 'GetAllVmStatsVDSCommand(HostName = virt2, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e'})' execution failed: Recovering from crash or Initializing 2017-10-01 14:49:36,424-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.PollVmStatsRefresher] (DefaultQuartzScheduler3) [6a80bf7c] Failed to fetch vms info for host 'virt2' - skipping VMs monitoring. 2017-10-01 14:49:40,326-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [102df1a8] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command GetCapabilitiesVDS failed: Recovering from crash or Initializing 2017-10-01 14:49:40,326-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler7) [102df1a8] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@7988582' 2017-10-01 14:49:40,329-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler7) [102df1a8] HostName = virt2 2017-10-01 14:49:40,329-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler7) [102df1a8] Command 'GetCapabilitiesVDSCommand(HostName = virt2, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vds='Host[virt2,a0ccfd11-3402-4eba-a119-4987561b738e]'})' execution failed: Recovering from crash or Initializing 2017-10-01 14:49:44,274-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [6e547a68] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command GetCapabilitiesVDS failed: Recovering from crash or Initializing 2017-10-01 14:49:44,274-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler9) [6e547a68] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@42ffb93a' 2017-10-01 14:49:44,276-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler9) [6e547a68] HostName = virt2 2017-10-01 14:49:44,276-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler9) [6e547a68] Command 'GetCapabilitiesVDSCommand(HostName = virt2, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vds='Host[virt2,a0ccfd11-3402-4eba-a119-4987561b738e]'})' execution failed: Recovering from crash or Initializing 2017-10-01 14:49:48,298-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [67da5c22] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command GetCapabilitiesVDS failed: Recovering from crash or Initializing 2017-10-01 14:49:48,298-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler10) [67da5c22] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@3793159a' 2017-10-01 14:49:48,298-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler10) [67da5c22] HostName = virt2 2017-10-01 14:49:48,298-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler10) [67da5c22] Command 'GetCapabilitiesVDSCommand(HostName = virt2, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vds='Host[virt2,a0ccfd11-3402-4eba-a119-4987561b738e]'})' execution failed: Recovering from crash or Initializing 2017-10-01 14:49:51,431-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [71eab7db] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command GetCapabilitiesVDS failed: Recovering from crash or Initializing 2017-10-01 14:49:51,431-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler7) [71eab7db] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@5cf632b5' 2017-10-01 14:49:51,431-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler7) [71eab7db] HostName = virt2 2017-10-01 14:49:51,431-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler7) [71eab7db] Command 'GetCapabilitiesVDSCommand(HostName = virt2, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vds='Host[virt2,a0ccfd11-3402-4eba-a119-4987561b738e]'})' execution failed: Recovering from crash or Initializing 2017-10-01 14:49:51,456-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler9) [4fc46bd1] Command 'GetAllVmStatsVDSCommand(HostName = virt2, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e'})' execution failed: Recovering from crash or Initializing 2017-10-01 14:49:51,456-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.PollVmStatsRefresher] (DefaultQuartzScheduler9) [4fc46bd1] Failed to fetch vms info for host 'virt2' - skipping VMs monitoring. 2017-10-01 14:49:54,762-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [7dc8ca5d] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 14:49:55,454-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [37b0083b] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command GetCapabilitiesVDS failed: Recovering from crash or Initializing 2017-10-01 14:49:55,454-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler6) [37b0083b] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@700a8777' 2017-10-01 14:49:55,454-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler6) [37b0083b] HostName = virt2 2017-10-01 14:49:55,454-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler6) [37b0083b] Command 'GetCapabilitiesVDSCommand(HostName = virt2, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vds='Host[virt2,a0ccfd11-3402-4eba-a119-4987561b738e]'})' execution failed: Recovering from crash or Initializing 2017-10-01 14:49:59,379-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [512f28f9] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command GetCapabilitiesVDS failed: Recovering from crash or Initializing 2017-10-01 14:49:59,379-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler6) [512f28f9] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@69b33612' 2017-10-01 14:49:59,379-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler6) [512f28f9] HostName = virt2 2017-10-01 14:49:59,379-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler6) [512f28f9] Command 'GetCapabilitiesVDSCommand(HostName = virt2, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vds='Host[virt2,a0ccfd11-3402-4eba-a119-4987561b738e]'})' execution failed: Recovering from crash or Initializing 2017-10-01 14:49:59,412-04 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-14) [] domain 'f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2:export' in problem 'PROBLEMATIC'. vds: 'virt4' 2017-10-01 14:49:59,413-04 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-14) [] domain '63d680c7-4c94-4b29-a6cb-f288a128b9bc:iso' in problem 'PROBLEMATIC'. vds: 'virt4' 2017-10-01 14:50:03,401-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [31be08c8] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command GetCapabilitiesVDS failed: Recovering from crash or Initializing 2017-10-01 14:50:03,401-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler4) [31be08c8] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@35f7d1a6' 2017-10-01 14:50:03,401-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler4) [31be08c8] HostName = virt2 2017-10-01 14:50:03,401-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler4) [31be08c8] Command 'GetCapabilitiesVDSCommand(HostName = virt2, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vds='Host[virt2,a0ccfd11-3402-4eba-a119-4987561b738e]'})' execution failed: Recovering from crash or Initializing 2017-10-01 14:50:06,461-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [670e62d6] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command GetCapabilitiesVDS failed: Recovering from crash or Initializing 2017-10-01 14:50:06,461-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler10) [670e62d6] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@983de94' 2017-10-01 14:50:06,461-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler10) [670e62d6] HostName = virt2 2017-10-01 14:50:06,461-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler10) [670e62d6] Command 'GetCapabilitiesVDSCommand(HostName = virt2, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vds='Host[virt2,a0ccfd11-3402-4eba-a119-4987561b738e]'})' execution failed: Recovering from crash or Initializing 2017-10-01 14:50:06,479-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler3) [59f88c5] Command 'GetAllVmStatsVDSCommand(HostName = virt2, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e'})' execution failed: Recovering from crash or Initializing 2017-10-01 14:50:06,480-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.PollVmStatsRefresher] (DefaultQuartzScheduler3) [59f88c5] Failed to fetch vms info for host 'virt2' - skipping VMs monitoring. 2017-10-01 14:50:08,814-04 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-21) [] domain 'c8653dba-63ec-4447-872c-0ffcc6af9c0f:data' in problem 'PROBLEMATIC'. vds: 'virt3' 2017-10-01 14:50:10,484-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [5e1e6428] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command GetCapabilitiesVDS failed: Recovering from crash or Initializing 2017-10-01 14:50:10,484-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler4) [5e1e6428] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@132d4c82' 2017-10-01 14:50:10,484-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler4) [5e1e6428] HostName = virt2 2017-10-01 14:50:10,484-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler4) [5e1e6428] Command 'GetCapabilitiesVDSCommand(HostName = virt2, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vds='Host[virt2,a0ccfd11-3402-4eba-a119-4987561b738e]'})' execution failed: Recovering from crash or Initializing 2017-10-01 14:50:14,494-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [36580d6e] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command GetCapabilitiesVDS failed: Recovering from crash or Initializing 2017-10-01 14:50:14,494-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler10) [36580d6e] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@4af01a44' 2017-10-01 14:50:14,494-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler10) [36580d6e] HostName = virt2 2017-10-01 14:50:14,494-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler10) [36580d6e] Command 'GetCapabilitiesVDSCommand(HostName = virt2, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vds='Host[virt2,a0ccfd11-3402-4eba-a119-4987561b738e]'})' execution failed: Recovering from crash or Initializing 2017-10-01 14:50:18,516-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [6bb7ef89] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command GetCapabilitiesVDS failed: Recovering from crash or Initializing 2017-10-01 14:50:18,516-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler5) [6bb7ef89] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@425d4dc' 2017-10-01 14:50:18,516-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler5) [6bb7ef89] HostName = virt2 2017-10-01 14:50:18,516-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler5) [6bb7ef89] Command 'GetCapabilitiesVDSCommand(HostName = virt2, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vds='Host[virt2,a0ccfd11-3402-4eba-a119-4987561b738e]'})' execution failed: Recovering from crash or Initializing 2017-10-01 14:50:21,512-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler7) [76a316e7] Command 'GetAllVmStatsVDSCommand(HostName = virt2, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e'})' execution failed: Recovering from crash or Initializing 2017-10-01 14:50:21,512-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.PollVmStatsRefresher] (DefaultQuartzScheduler7) [76a316e7] Failed to fetch vms info for host 'virt2' - skipping VMs monitoring. 2017-10-01 14:50:22,538-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [5f846e52] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command GetCapabilitiesVDS failed: Recovering from crash or Initializing 2017-10-01 14:50:22,538-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler8) [5f846e52] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@394965ea' 2017-10-01 14:50:22,538-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler8) [5f846e52] HostName = virt2 2017-10-01 14:50:22,538-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler8) [5f846e52] Command 'GetCapabilitiesVDSCommand(HostName = virt2, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vds='Host[virt2,a0ccfd11-3402-4eba-a119-4987561b738e]'})' execution failed: Recovering from crash or Initializing 2017-10-01 14:50:26,562-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [5b9140d9] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command GetCapabilitiesVDS failed: Recovering from crash or Initializing 2017-10-01 14:50:26,562-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler4) [5b9140d9] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@40455a8a' 2017-10-01 14:50:26,562-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler4) [5b9140d9] HostName = virt2 2017-10-01 14:50:26,562-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler4) [5b9140d9] Command 'GetCapabilitiesVDSCommand(HostName = virt2, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vds='Host[virt2,a0ccfd11-3402-4eba-a119-4987561b738e]'})' execution failed: Recovering from crash or Initializing 2017-10-01 14:50:29,603-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [6ead1ec5] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command GetCapabilitiesVDS failed: Recovering from crash or Initializing 2017-10-01 14:50:29,603-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler4) [6ead1ec5] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@42aa525d' 2017-10-01 14:50:29,603-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler4) [6ead1ec5] HostName = virt2 2017-10-01 14:50:29,603-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler4) [6ead1ec5] Command 'GetCapabilitiesVDSCommand(HostName = virt2, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vds='Host[virt2,a0ccfd11-3402-4eba-a119-4987561b738e]'})' execution failed: Recovering from crash or Initializing 2017-10-01 14:50:33,647-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [6ed94867] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command GetCapabilitiesVDS failed: Recovering from crash or Initializing 2017-10-01 14:50:33,647-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler10) [6ed94867] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@68db3145' 2017-10-01 14:50:33,647-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler10) [6ed94867] HostName = virt2 2017-10-01 14:50:33,647-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler10) [6ed94867] Command 'GetCapabilitiesVDSCommand(HostName = virt2, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vds='Host[virt2,a0ccfd11-3402-4eba-a119-4987561b738e]'})' execution failed: Recovering from crash or Initializing 2017-10-01 14:50:36,535-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler9) [28d775fc] Command 'GetAllVmStatsVDSCommand(HostName = virt2, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e'})' execution failed: Recovering from crash or Initializing 2017-10-01 14:50:36,535-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.PollVmStatsRefresher] (DefaultQuartzScheduler9) [28d775fc] Failed to fetch vms info for host 'virt2' - skipping VMs monitoring. 2017-10-01 14:50:37,670-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [395322f1] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command GetCapabilitiesVDS failed: Recovering from crash or Initializing 2017-10-01 14:50:37,670-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler6) [395322f1] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@2291bcce' 2017-10-01 14:50:37,670-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler6) [395322f1] HostName = virt2 2017-10-01 14:50:37,670-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler6) [395322f1] Command 'GetCapabilitiesVDSCommand(HostName = virt2, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vds='Host[virt2,a0ccfd11-3402-4eba-a119-4987561b738e]'})' execution failed: Recovering from crash or Initializing 2017-10-01 14:50:41,692-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [5432ee04] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command GetCapabilitiesVDS failed: Recovering from crash or Initializing 2017-10-01 14:50:41,692-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler7) [5432ee04] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@19efa96e' 2017-10-01 14:50:41,692-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler7) [5432ee04] HostName = virt2 2017-10-01 14:50:41,692-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler7) [5432ee04] Command 'GetCapabilitiesVDSCommand(HostName = virt2, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vds='Host[virt2,a0ccfd11-3402-4eba-a119-4987561b738e]'})' execution failed: Recovering from crash or Initializing 2017-10-01 14:50:45,715-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [3c649a9b] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command GetCapabilitiesVDS failed: Recovering from crash or Initializing 2017-10-01 14:50:45,715-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler3) [3c649a9b] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@e1e77aa' 2017-10-01 14:50:45,715-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler3) [3c649a9b] HostName = virt2 2017-10-01 14:50:45,715-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler3) [3c649a9b] Command 'GetCapabilitiesVDSCommand(HostName = virt2, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vds='Host[virt2,a0ccfd11-3402-4eba-a119-4987561b738e]'})' execution failed: Recovering from crash or Initializing 2017-10-01 14:50:49,737-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [1ab059e6] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command GetCapabilitiesVDS failed: Recovering from crash or Initializing 2017-10-01 14:50:49,737-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler3) [1ab059e6] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@549e577c' 2017-10-01 14:50:49,737-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler3) [1ab059e6] HostName = virt2 2017-10-01 14:50:49,737-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler3) [1ab059e6] Command 'GetCapabilitiesVDSCommand(HostName = virt2, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vds='Host[virt2,a0ccfd11-3402-4eba-a119-4987561b738e]'})' execution failed: Recovering from crash or Initializing 2017-10-01 14:50:51,566-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler3) [290885ff] Command 'GetAllVmStatsVDSCommand(HostName = virt2, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e'})' execution failed: Recovering from crash or Initializing 2017-10-01 14:50:51,566-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.PollVmStatsRefresher] (DefaultQuartzScheduler3) [290885ff] Failed to fetch vms info for host 'virt2' - skipping VMs monitoring. 2017-10-01 14:50:52,761-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3531151] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 14:50:52,761-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [3531151] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 14:50:52,761-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [3531151] HostName = virt4 2017-10-01 14:50:52,761-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [3531151] Command 'ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 14:50:52,761-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [3531151] FINISH, ConnectStoragePoolVDSCommand, log id: 229e55d6 2017-10-01 14:50:52,761-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler2) [3531151] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 14:50:52,861-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [131d6d71] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command GetCapabilitiesVDS failed: Recovering from crash or Initializing 2017-10-01 14:50:52,861-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler8) [131d6d71] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@1dac731e' 2017-10-01 14:50:52,861-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler8) [131d6d71] HostName = virt2 2017-10-01 14:50:52,861-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler8) [131d6d71] Command 'GetCapabilitiesVDSCommand(HostName = virt2, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vds='Host[virt2,a0ccfd11-3402-4eba-a119-4987561b738e]'})' execution failed: Recovering from crash or Initializing 2017-10-01 14:50:54,781-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [6d9b463c] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 14:50:56,883-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [76201e95] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command GetCapabilitiesVDS failed: Recovering from crash or Initializing 2017-10-01 14:50:56,883-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler10) [76201e95] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@5752e10a' 2017-10-01 14:50:56,883-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler10) [76201e95] HostName = virt2 2017-10-01 14:50:56,883-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler10) [76201e95] Command 'GetCapabilitiesVDSCommand(HostName = virt2, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vds='Host[virt2,a0ccfd11-3402-4eba-a119-4987561b738e]'})' execution failed: Recovering from crash or Initializing 2017-10-01 14:51:00,905-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [7a5df9f9] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command GetCapabilitiesVDS failed: Recovering from crash or Initializing 2017-10-01 14:51:00,905-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler10) [7a5df9f9] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@24c14f68' 2017-10-01 14:51:00,905-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler10) [7a5df9f9] HostName = virt2 2017-10-01 14:51:00,905-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler10) [7a5df9f9] Command 'GetCapabilitiesVDSCommand(HostName = virt2, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vds='Host[virt2,a0ccfd11-3402-4eba-a119-4987561b738e]'})' execution failed: Recovering from crash or Initializing 2017-10-01 14:51:04,937-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [9396ac1] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command GetCapabilitiesVDS failed: Recovering from crash or Initializing 2017-10-01 14:51:04,937-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler7) [9396ac1] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@9daa95b' 2017-10-01 14:51:04,937-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler7) [9396ac1] HostName = virt2 2017-10-01 14:51:04,937-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler7) [9396ac1] Command 'GetCapabilitiesVDSCommand(HostName = virt2, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vds='Host[virt2,a0ccfd11-3402-4eba-a119-4987561b738e]'})' execution failed: Recovering from crash or Initializing 2017-10-01 14:51:06,591-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler8) [4c7508e2] Command 'GetAllVmStatsVDSCommand(HostName = virt2, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e'})' execution failed: Recovering from crash or Initializing 2017-10-01 14:51:06,591-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.PollVmStatsRefresher] (DefaultQuartzScheduler8) [4c7508e2] Failed to fetch vms info for host 'virt2' - skipping VMs monitoring. 2017-10-01 14:51:08,965-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [149060ec] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command GetCapabilitiesVDS failed: Recovering from crash or Initializing 2017-10-01 14:51:08,965-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler3) [149060ec] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@47e11b35' 2017-10-01 14:51:08,965-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler3) [149060ec] HostName = virt2 2017-10-01 14:51:08,965-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler3) [149060ec] Command 'GetCapabilitiesVDSCommand(HostName = virt2, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vds='Host[virt2,a0ccfd11-3402-4eba-a119-4987561b738e]'})' execution failed: Recovering from crash or Initializing 2017-10-01 14:51:12,987-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [49232a8b] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command GetCapabilitiesVDS failed: Recovering from crash or Initializing 2017-10-01 14:51:12,987-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler10) [49232a8b] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@6f60151a' 2017-10-01 14:51:12,987-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler10) [49232a8b] HostName = virt2 2017-10-01 14:51:12,987-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler10) [49232a8b] Command 'GetCapabilitiesVDSCommand(HostName = virt2, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vds='Host[virt2,a0ccfd11-3402-4eba-a119-4987561b738e]'})' execution failed: Recovering from crash or Initializing 2017-10-01 14:51:17,010-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [1c2c1d11] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command GetCapabilitiesVDS failed: Recovering from crash or Initializing 2017-10-01 14:51:17,010-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler8) [1c2c1d11] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@41b01e01' 2017-10-01 14:51:17,010-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler8) [1c2c1d11] HostName = virt2 2017-10-01 14:51:17,010-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler8) [1c2c1d11] Command 'GetCapabilitiesVDSCommand(HostName = virt2, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vds='Host[virt2,a0ccfd11-3402-4eba-a119-4987561b738e]'})' execution failed: Recovering from crash or Initializing 2017-10-01 14:51:21,032-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [2dc666d1] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command GetCapabilitiesVDS failed: Recovering from crash or Initializing 2017-10-01 14:51:21,032-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler3) [2dc666d1] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@208e7b4d' 2017-10-01 14:51:21,032-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler3) [2dc666d1] HostName = virt2 2017-10-01 14:51:21,032-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler3) [2dc666d1] Command 'GetCapabilitiesVDSCommand(HostName = virt2, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vds='Host[virt2,a0ccfd11-3402-4eba-a119-4987561b738e]'})' execution failed: Recovering from crash or Initializing 2017-10-01 14:51:21,622-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler5) [19f56a42] Command 'GetAllVmStatsVDSCommand(HostName = virt2, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e'})' execution failed: Recovering from crash or Initializing 2017-10-01 14:51:21,622-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.PollVmStatsRefresher] (DefaultQuartzScheduler5) [19f56a42] Failed to fetch vms info for host 'virt2' - skipping VMs monitoring. 2017-10-01 14:51:25,052-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [63fe5a1f] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command GetCapabilitiesVDS failed: Recovering from crash or Initializing 2017-10-01 14:51:25,052-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler9) [63fe5a1f] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@1c8bc317' 2017-10-01 14:51:25,052-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler9) [63fe5a1f] HostName = virt2 2017-10-01 14:51:25,052-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler9) [63fe5a1f] Command 'GetCapabilitiesVDSCommand(HostName = virt2, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vds='Host[virt2,a0ccfd11-3402-4eba-a119-4987561b738e]'})' execution failed: Recovering from crash or Initializing 2017-10-01 14:51:29,074-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [42f64384] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command GetCapabilitiesVDS failed: Recovering from crash or Initializing 2017-10-01 14:51:29,074-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler10) [42f64384] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@59af109a' 2017-10-01 14:51:29,074-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler10) [42f64384] HostName = virt2 2017-10-01 14:51:29,074-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler10) [42f64384] Command 'GetCapabilitiesVDSCommand(HostName = virt2, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vds='Host[virt2,a0ccfd11-3402-4eba-a119-4987561b738e]'})' execution failed: Recovering from crash or Initializing 2017-10-01 14:51:33,043-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [2c445a8a] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command GetCapabilitiesVDS failed: Recovering from crash or Initializing 2017-10-01 14:51:33,043-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler9) [2c445a8a] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@3b1b8c1d' 2017-10-01 14:51:33,043-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler9) [2c445a8a] HostName = virt2 2017-10-01 14:51:33,043-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler9) [2c445a8a] Command 'GetCapabilitiesVDSCommand(HostName = virt2, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vds='Host[virt2,a0ccfd11-3402-4eba-a119-4987561b738e]'})' execution failed: Recovering from crash or Initializing 2017-10-01 14:51:36,628-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [31da3136] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command GetCapabilitiesVDS failed: Recovering from crash or Initializing 2017-10-01 14:51:36,628-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler4) [31da3136] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@76de3230' 2017-10-01 14:51:36,628-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler4) [31da3136] HostName = virt2 2017-10-01 14:51:36,628-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler4) [31da3136] Command 'GetCapabilitiesVDSCommand(HostName = virt2, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vds='Host[virt2,a0ccfd11-3402-4eba-a119-4987561b738e]'})' execution failed: Recovering from crash or Initializing 2017-10-01 14:51:36,653-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler3) [34bd237] Command 'GetAllVmStatsVDSCommand(HostName = virt2, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e'})' execution failed: Recovering from crash or Initializing 2017-10-01 14:51:36,653-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.PollVmStatsRefresher] (DefaultQuartzScheduler3) [34bd237] Failed to fetch vms info for host 'virt2' - skipping VMs monitoring. 2017-10-01 14:51:40,650-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [7b34294c] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command GetCapabilitiesVDS failed: Recovering from crash or Initializing 2017-10-01 14:51:40,650-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler8) [7b34294c] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@7b884e50' 2017-10-01 14:51:40,650-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler8) [7b34294c] HostName = virt2 2017-10-01 14:51:40,650-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler8) [7b34294c] Command 'GetCapabilitiesVDSCommand(HostName = virt2, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vds='Host[virt2,a0ccfd11-3402-4eba-a119-4987561b738e]'})' execution failed: Recovering from crash or Initializing 2017-10-01 14:51:44,673-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [7e97df80] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command GetCapabilitiesVDS failed: Recovering from crash or Initializing 2017-10-01 14:51:44,673-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler7) [7e97df80] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@5053f0c2' 2017-10-01 14:51:44,673-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler7) [7e97df80] HostName = virt2 2017-10-01 14:51:44,673-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler7) [7e97df80] Command 'GetCapabilitiesVDSCommand(HostName = virt2, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vds='Host[virt2,a0ccfd11-3402-4eba-a119-4987561b738e]'})' execution failed: Recovering from crash or Initializing 2017-10-01 14:51:48,158-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [40ae499e] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command GetCapabilitiesVDS failed: Recovering from crash or Initializing 2017-10-01 14:51:48,158-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler6) [40ae499e] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@2b189848' 2017-10-01 14:51:48,158-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler6) [40ae499e] HostName = virt2 2017-10-01 14:51:48,158-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler6) [40ae499e] Command 'GetCapabilitiesVDSCommand(HostName = virt2, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vds='Host[virt2,a0ccfd11-3402-4eba-a119-4987561b738e]'})' execution failed: Recovering from crash or Initializing 2017-10-01 14:51:51,658-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [36a1c459] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command GetCapabilitiesVDS failed: Recovering from crash or Initializing 2017-10-01 14:51:51,658-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler4) [36a1c459] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@6a7a8f55' 2017-10-01 14:51:51,658-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler4) [36a1c459] HostName = virt2 2017-10-01 14:51:51,658-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler4) [36a1c459] Command 'GetCapabilitiesVDSCommand(HostName = virt2, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vds='Host[virt2,a0ccfd11-3402-4eba-a119-4987561b738e]'})' execution failed: Recovering from crash or Initializing 2017-10-01 14:51:51,678-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler9) [11dd12e6] Command 'GetAllVmStatsVDSCommand(HostName = virt2, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e'})' execution failed: Recovering from crash or Initializing 2017-10-01 14:51:51,678-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.PollVmStatsRefresher] (DefaultQuartzScheduler9) [11dd12e6] Failed to fetch vms info for host 'virt2' - skipping VMs monitoring. 2017-10-01 14:51:54,800-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [45460b55] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 14:51:55,680-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [5195d71f] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command GetCapabilitiesVDS failed: Recovering from crash or Initializing 2017-10-01 14:51:55,680-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler3) [5195d71f] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@6f856252' 2017-10-01 14:51:55,680-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler3) [5195d71f] HostName = virt2 2017-10-01 14:51:55,680-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler3) [5195d71f] Command 'GetCapabilitiesVDSCommand(HostName = virt2, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vds='Host[virt2,a0ccfd11-3402-4eba-a119-4987561b738e]'})' execution failed: Recovering from crash or Initializing 2017-10-01 14:51:59,703-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [51e225e8] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command GetCapabilitiesVDS failed: Recovering from crash or Initializing 2017-10-01 14:51:59,703-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler6) [51e225e8] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@402dbdc1' 2017-10-01 14:51:59,703-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler6) [51e225e8] HostName = virt2 2017-10-01 14:51:59,703-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler6) [51e225e8] Command 'GetCapabilitiesVDSCommand(HostName = virt2, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vds='Host[virt2,a0ccfd11-3402-4eba-a119-4987561b738e]'})' execution failed: Recovering from crash or Initializing 2017-10-01 14:52:03,299-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [2c065ee1] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command GetCapabilitiesVDS failed: Recovering from crash or Initializing 2017-10-01 14:52:03,299-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler1) [2c065ee1] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@7cede130' 2017-10-01 14:52:03,299-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler1) [2c065ee1] HostName = virt2 2017-10-01 14:52:03,299-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler1) [2c065ee1] Command 'GetCapabilitiesVDSCommand(HostName = virt2, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vds='Host[virt2,a0ccfd11-3402-4eba-a119-4987561b738e]'})' execution failed: Recovering from crash or Initializing 2017-10-01 14:52:06,683-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [5c58fcc9] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command GetCapabilitiesVDS failed: Recovering from crash or Initializing 2017-10-01 14:52:06,683-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler8) [5c58fcc9] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@37a6deaa' 2017-10-01 14:52:06,683-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler8) [5c58fcc9] HostName = virt2 2017-10-01 14:52:06,683-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler8) [5c58fcc9] Command 'GetCapabilitiesVDSCommand(HostName = virt2, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vds='Host[virt2,a0ccfd11-3402-4eba-a119-4987561b738e]'})' execution failed: Recovering from crash or Initializing 2017-10-01 14:52:06,709-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler5) [688d71db] Command 'GetAllVmStatsVDSCommand(HostName = virt2, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e'})' execution failed: Recovering from crash or Initializing 2017-10-01 14:52:06,709-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.PollVmStatsRefresher] (DefaultQuartzScheduler5) [688d71db] Failed to fetch vms info for host 'virt2' - skipping VMs monitoring. 2017-10-01 14:52:10,706-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [3a2f2e63] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command GetCapabilitiesVDS failed: Recovering from crash or Initializing 2017-10-01 14:52:10,706-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler8) [3a2f2e63] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@49a04b7c' 2017-10-01 14:52:10,706-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler8) [3a2f2e63] HostName = virt2 2017-10-01 14:52:10,706-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler8) [3a2f2e63] Command 'GetCapabilitiesVDSCommand(HostName = virt2, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vds='Host[virt2,a0ccfd11-3402-4eba-a119-4987561b738e]'})' execution failed: Recovering from crash or Initializing 2017-10-01 14:52:14,729-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [119b2870] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command GetCapabilitiesVDS failed: Recovering from crash or Initializing 2017-10-01 14:52:14,729-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler3) [119b2870] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@4a8037b1' 2017-10-01 14:52:14,729-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler3) [119b2870] HostName = virt2 2017-10-01 14:52:14,729-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler3) [119b2870] Command 'GetCapabilitiesVDSCommand(HostName = virt2, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vds='Host[virt2,a0ccfd11-3402-4eba-a119-4987561b738e]'})' execution failed: Recovering from crash or Initializing 2017-10-01 14:52:18,751-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [5cbc2a18] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command GetCapabilitiesVDS failed: Recovering from crash or Initializing 2017-10-01 14:52:18,751-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler9) [5cbc2a18] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@614f6624' 2017-10-01 14:52:18,751-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler9) [5cbc2a18] HostName = virt2 2017-10-01 14:52:18,751-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler9) [5cbc2a18] Command 'GetCapabilitiesVDSCommand(HostName = virt2, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vds='Host[virt2,a0ccfd11-3402-4eba-a119-4987561b738e]'})' execution failed: Recovering from crash or Initializing 2017-10-01 14:52:21,739-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler4) [8ec1ce9] Command 'GetAllVmStatsVDSCommand(HostName = virt2, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e'})' execution failed: Recovering from crash or Initializing 2017-10-01 14:52:21,739-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.PollVmStatsRefresher] (DefaultQuartzScheduler4) [8ec1ce9] Failed to fetch vms info for host 'virt2' - skipping VMs monitoring. 2017-10-01 14:52:22,773-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [403018d9] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command GetCapabilitiesVDS failed: Recovering from crash or Initializing 2017-10-01 14:52:22,773-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler7) [403018d9] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@170c544b' 2017-10-01 14:52:22,773-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler7) [403018d9] HostName = virt2 2017-10-01 14:52:22,773-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler7) [403018d9] Command 'GetCapabilitiesVDSCommand(HostName = virt2, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vds='Host[virt2,a0ccfd11-3402-4eba-a119-4987561b738e]'})' execution failed: Recovering from crash or Initializing 2017-10-01 14:52:26,795-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [1f0ae122] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command GetCapabilitiesVDS failed: Recovering from crash or Initializing 2017-10-01 14:52:26,795-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler6) [1f0ae122] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@2372c53a' 2017-10-01 14:52:26,795-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler6) [1f0ae122] HostName = virt2 2017-10-01 14:52:26,795-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler6) [1f0ae122] Command 'GetCapabilitiesVDSCommand(HostName = virt2, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vds='Host[virt2,a0ccfd11-3402-4eba-a119-4987561b738e]'})' execution failed: Recovering from crash or Initializing 2017-10-01 14:52:30,817-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [47aa0e3] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command GetCapabilitiesVDS failed: Recovering from crash or Initializing 2017-10-01 14:52:30,817-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler5) [47aa0e3] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@270dc3cf' 2017-10-01 14:52:30,817-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler5) [47aa0e3] HostName = virt2 2017-10-01 14:52:30,817-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler5) [47aa0e3] Command 'GetCapabilitiesVDSCommand(HostName = virt2, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vds='Host[virt2,a0ccfd11-3402-4eba-a119-4987561b738e]'})' execution failed: Recovering from crash or Initializing 2017-10-01 14:52:32,839-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler8) [4cd00a09] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: a0ccfd11-3402-4eba-a119-4987561b738e Type: VDS 2017-10-01 14:52:32,854-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [4cd00a09] START, SetVdsStatusVDSCommand(HostName = virt2, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', status='NonOperational', nonOperationalReason='TIMEOUT_RECOVERING_FROM_CRASH', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 366c35bf 2017-10-01 14:52:32,854-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [4cd00a09] VDS 'virt2' is spm and moved from up calling resetIrs. 2017-10-01 14:52:32,854-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (DefaultQuartzScheduler8) [4cd00a09] START, ResetIrsVDSCommand( ResetIrsVDSCommandParameters:{runAsync='true', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', ignoreFailoverLimit='false', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', ignoreStopFailed='false'}), log id: 41325306 2017-10-01 14:52:32,869-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (DefaultQuartzScheduler8) [4cd00a09] START, SpmStopVDSCommand(HostName = virt2, SpmStopVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'}), log id: 4c87303 2017-10-01 14:52:33,878-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [4cd00a09] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command HSMGetAllTasksStatusesVDS failed: Recovering from crash or Initializing 2017-10-01 14:52:33,878-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler8) [4cd00a09] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand' return value ' 2017-10-01 14:52:33,878-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler8) [4cd00a09] HostName = virt2 2017-10-01 14:52:33,878-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler8) [4cd00a09] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = virt2, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e'})' execution failed: Recovering from crash or Initializing 2017-10-01 14:52:33,878-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (DefaultQuartzScheduler8) [4cd00a09] SpmStopVDSCommand::Stopping SPM on vds 'virt2', pool id 'aca61eac-ed8a-484b-bbee-3645148f6405' 2017-10-01 14:52:34,885-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [4cd00a09] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command SpmStopVDS failed: Recovering from crash or Initializing 2017-10-01 14:52:34,886-04 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (DefaultQuartzScheduler8) [4cd00a09] Could not stop spm of pool 'aca61eac-ed8a-484b-bbee-3645148f6405' on vds 'a0ccfd11-3402-4eba-a119-4987561b738e': Recovering from crash or Initializing 2017-10-01 14:52:34,886-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (DefaultQuartzScheduler8) [4cd00a09] FINISH, SpmStopVDSCommand, log id: 4c87303 2017-10-01 14:52:34,886-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (DefaultQuartzScheduler8) [4cd00a09] FINISH, ResetIrsVDSCommand, log id: 41325306 2017-10-01 14:52:34,888-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [4cd00a09] FINISH, SetVdsStatusVDSCommand, log id: 366c35bf 2017-10-01 14:52:34,920-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [4cd00a09] EVENT_ID: VDS_RECOVER_FAILED(19), Correlation ID: 4cd00a09, Job ID: 51f977da-b75d-4015-a771-6565ead9a9eb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host virt2 failed to recover. 2017-10-01 14:52:36,770-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler7) [36bb83c7] Command 'GetAllVmStatsVDSCommand(HostName = virt2, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e'})' execution failed: Recovering from crash or Initializing 2017-10-01 14:52:36,770-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.PollVmStatsRefresher] (DefaultQuartzScheduler7) [36bb83c7] Failed to fetch vms info for host 'virt2' - skipping VMs monitoring. 2017-10-01 14:52:41,950-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [5219bccf] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command GetStatsVDS failed: Recovering from crash or Initializing 2017-10-01 14:52:41,950-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStatsVDSCommand] (DefaultQuartzScheduler8) [5219bccf] Command 'GetStatsVDSCommand(HostName = virt2, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vds='Host[virt2,a0ccfd11-3402-4eba-a119-4987561b738e]'})' execution failed: Recovering from crash or Initializing 2017-10-01 14:52:41,950-04 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [5219bccf] Failed getting vds stats, host='virt2'(a0ccfd11-3402-4eba-a119-4987561b738e): org.ovirt.engine.core.vdsbroker.vdsbroker.VDSRecoveringException: Recovering from crash or Initializing 2017-10-01 14:52:51,796-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler4) [4b0aa2fa] Command 'GetAllVmStatsVDSCommand(HostName = virt2, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e'})' execution failed: Recovering from crash or Initializing 2017-10-01 14:52:51,796-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.PollVmStatsRefresher] (DefaultQuartzScheduler4) [4b0aa2fa] Failed to fetch vms info for host 'virt2' - skipping VMs monitoring. 2017-10-01 14:52:54,826-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [33e99aad] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 14:52:58,037-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [2deb2c73] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command GetStatsVDS failed: Recovering from crash or Initializing 2017-10-01 14:52:58,037-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStatsVDSCommand] (DefaultQuartzScheduler10) [2deb2c73] Command 'GetStatsVDSCommand(HostName = virt2, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vds='Host[virt2,a0ccfd11-3402-4eba-a119-4987561b738e]'})' execution failed: Recovering from crash or Initializing 2017-10-01 14:52:58,038-04 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [2deb2c73] Failed getting vds stats, host='virt2'(a0ccfd11-3402-4eba-a119-4987561b738e): org.ovirt.engine.core.vdsbroker.vdsbroker.VDSRecoveringException: Recovering from crash or Initializing 2017-10-01 14:53:06,826-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler8) [75db2a46] Command 'GetAllVmStatsVDSCommand(HostName = virt2, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e'})' execution failed: Recovering from crash or Initializing 2017-10-01 14:53:06,826-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.PollVmStatsRefresher] (DefaultQuartzScheduler8) [75db2a46] Failed to fetch vms info for host 'virt2' - skipping VMs monitoring. 2017-10-01 14:53:14,123-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [3503a085] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command GetStatsVDS failed: Recovering from crash or Initializing 2017-10-01 14:53:14,123-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStatsVDSCommand] (DefaultQuartzScheduler8) [3503a085] Command 'GetStatsVDSCommand(HostName = virt2, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vds='Host[virt2,a0ccfd11-3402-4eba-a119-4987561b738e]'})' execution failed: Recovering from crash or Initializing 2017-10-01 14:53:14,123-04 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [3503a085] Failed getting vds stats, host='virt2'(a0ccfd11-3402-4eba-a119-4987561b738e): org.ovirt.engine.core.vdsbroker.vdsbroker.VDSRecoveringException: Recovering from crash or Initializing 2017-10-01 14:53:21,849-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler3) [6dc5f6d4] Command 'GetAllVmStatsVDSCommand(HostName = virt2, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e'})' execution failed: Recovering from crash or Initializing 2017-10-01 14:53:21,849-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.PollVmStatsRefresher] (DefaultQuartzScheduler3) [6dc5f6d4] Failed to fetch vms info for host 'virt2' - skipping VMs monitoring. 2017-10-01 14:53:29,252-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [502e3f55] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command GetStatsVDS failed: Recovering from crash or Initializing 2017-10-01 14:53:29,252-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStatsVDSCommand] (DefaultQuartzScheduler7) [502e3f55] Command 'GetStatsVDSCommand(HostName = virt2, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vds='Host[virt2,a0ccfd11-3402-4eba-a119-4987561b738e]'})' execution failed: Recovering from crash or Initializing 2017-10-01 14:53:29,252-04 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [502e3f55] Failed getting vds stats, host='virt2'(a0ccfd11-3402-4eba-a119-4987561b738e): org.ovirt.engine.core.vdsbroker.vdsbroker.VDSRecoveringException: Recovering from crash or Initializing 2017-10-01 14:53:36,880-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler8) [399ffd83] Command 'GetAllVmStatsVDSCommand(HostName = virt2, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e'})' execution failed: Recovering from crash or Initializing 2017-10-01 14:53:36,880-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.PollVmStatsRefresher] (DefaultQuartzScheduler8) [399ffd83] Failed to fetch vms info for host 'virt2' - skipping VMs monitoring. 2017-10-01 14:53:44,364-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [5d035b3] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command GetStatsVDS failed: Recovering from crash or Initializing 2017-10-01 14:53:44,364-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStatsVDSCommand] (DefaultQuartzScheduler1) [5d035b3] Command 'GetStatsVDSCommand(HostName = virt2, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vds='Host[virt2,a0ccfd11-3402-4eba-a119-4987561b738e]'})' execution failed: Recovering from crash or Initializing 2017-10-01 14:53:44,364-04 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [5d035b3] Failed getting vds stats, host='virt2'(a0ccfd11-3402-4eba-a119-4987561b738e): org.ovirt.engine.core.vdsbroker.vdsbroker.VDSRecoveringException: Recovering from crash or Initializing 2017-10-01 14:53:51,905-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler3) [5000c187] Command 'GetAllVmStatsVDSCommand(HostName = virt2, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e'})' execution failed: Recovering from crash or Initializing 2017-10-01 14:53:51,905-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.PollVmStatsRefresher] (DefaultQuartzScheduler3) [5000c187] Failed to fetch vms info for host 'virt2' - skipping VMs monitoring. 2017-10-01 14:53:53,325-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3531151] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command SpmStatusVDS failed: Message timeout which can be caused by communication issues 2017-10-01 14:53:53,325-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler2) [3531151] Command 'SpmStatusVDSCommand(HostName = virt4, SpmStatusVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues 2017-10-01 14:53:53,325-04 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (org.ovirt.thread.pool-6-thread-19) [3531151] Host 'virt4' is not responding. It will stay in Connecting state for a grace period of 64 seconds and after that an attempt to fence the host will be issued. 2017-10-01 14:53:53,325-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler2) [3531151] Irs placed on server '5ae29ed2-9c0f-4754-8df3-b43d14726846' failed. Proceed Failover 2017-10-01 14:53:53,345-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-19) [3531151] EVENT_ID: VDS_HOST_NOT_RESPONDING_CONNECTING(9,008), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host virt4 is not responding. It will stay in Connecting state for a grace period of 64 seconds and after that an attempt to fence the host will be issued. 2017-10-01 14:53:53,384-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [3531151] START, ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 2bcc2b36 2017-10-01 14:53:53,384-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [3531151] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 14:53:54,845-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [13b2629] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 14:53:56,164-04 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to virt4-mgmt/192.168.0.37 2017-10-01 14:53:58,382-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [544d8f24] START, GetHardwareInfoVDSCommand(HostName = virt4, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vds='Host[virt4,5ae29ed2-9c0f-4754-8df3-b43d14726846]'}), log id: 718825af 2017-10-01 14:53:59,385-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [544d8f24] FINISH, GetHardwareInfoVDSCommand, log id: 718825af 2017-10-01 14:53:59,474-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [45487c1f] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command GetStatsVDS failed: Recovering from crash or Initializing 2017-10-01 14:53:59,475-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStatsVDSCommand] (DefaultQuartzScheduler5) [45487c1f] Command 'GetStatsVDSCommand(HostName = virt2, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vds='Host[virt2,a0ccfd11-3402-4eba-a119-4987561b738e]'})' execution failed: Recovering from crash or Initializing 2017-10-01 14:53:59,475-04 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [45487c1f] Failed getting vds stats, host='virt2'(a0ccfd11-3402-4eba-a119-4987561b738e): org.ovirt.engine.core.vdsbroker.vdsbroker.VDSRecoveringException: Recovering from crash or Initializing 2017-10-01 14:53:59,485-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [7deabfea] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5ae29ed2-9c0f-4754-8df3-b43d14726846 Type: VDS 2017-10-01 14:53:59,512-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [3eabca18] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: aca61eac-ed8a-484b-bbee-3645148f6405 Type: StoragePool 2017-10-01 14:53:59,520-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [78306a7] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: aca61eac-ed8a-484b-bbee-3645148f6405 Type: StoragePool 2017-10-01 14:53:59,592-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [78306a7] START, ConnectStorageServerVDSCommand(HostName = virt4, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', storageType='NFS', connectionList='[StorageServerConnections:{id='5e557659-6817-40fa-9a55-7458125404a6', connection='virtfs-fs:/nfs/data', iqn='null', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='13cf1e67-9961-4a25-a41f-cc76b4ab793d', connection='virtfs-fs:/nfs/iso', iqn='null', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='417fdd51-dbcd-41d9-9a05-ef4d9f485d13', connection='virtfs-fs:/nfs/export', iqn='null', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 16f9fc45 2017-10-01 14:54:06,937-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler1) [71b46a0b] Command 'GetAllVmStatsVDSCommand(HostName = virt2, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e'})' execution failed: Recovering from crash or Initializing 2017-10-01 14:54:06,937-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.PollVmStatsRefresher] (DefaultQuartzScheduler1) [71b46a0b] Failed to fetch vms info for host 'virt2' - skipping VMs monitoring. 2017-10-01 14:54:14,581-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [2a44e72e] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command GetStatsVDS failed: Recovering from crash or Initializing 2017-10-01 14:54:14,581-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStatsVDSCommand] (DefaultQuartzScheduler1) [2a44e72e] Command 'GetStatsVDSCommand(HostName = virt2, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vds='Host[virt2,a0ccfd11-3402-4eba-a119-4987561b738e]'})' execution failed: Recovering from crash or Initializing 2017-10-01 14:54:14,581-04 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [2a44e72e] Failed getting vds stats, host='virt2'(a0ccfd11-3402-4eba-a119-4987561b738e): org.ovirt.engine.core.vdsbroker.vdsbroker.VDSRecoveringException: Recovering from crash or Initializing 2017-10-01 14:54:21,969-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler3) [2146ddcd] Command 'GetAllVmStatsVDSCommand(HostName = virt2, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e'})' execution failed: Recovering from crash or Initializing 2017-10-01 14:54:21,969-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.PollVmStatsRefresher] (DefaultQuartzScheduler3) [2146ddcd] Failed to fetch vms info for host 'virt2' - skipping VMs monitoring. 2017-10-01 14:54:30,676-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2297468] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command GetStatsVDS failed: Recovering from crash or Initializing 2017-10-01 14:54:30,676-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStatsVDSCommand] (DefaultQuartzScheduler8) [2297468] Command 'GetStatsVDSCommand(HostName = virt2, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vds='Host[virt2,a0ccfd11-3402-4eba-a119-4987561b738e]'})' execution failed: Recovering from crash or Initializing 2017-10-01 14:54:30,676-04 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [2297468] Failed getting vds stats, host='virt2'(a0ccfd11-3402-4eba-a119-4987561b738e): org.ovirt.engine.core.vdsbroker.vdsbroker.VDSRecoveringException: Recovering from crash or Initializing 2017-10-01 14:54:36,994-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler4) [56e4832d] Command 'GetAllVmStatsVDSCommand(HostName = virt2, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e'})' execution failed: Recovering from crash or Initializing 2017-10-01 14:54:36,995-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.PollVmStatsRefresher] (DefaultQuartzScheduler4) [56e4832d] Failed to fetch vms info for host 'virt2' - skipping VMs monitoring. 2017-10-01 14:54:45,804-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [25ac73de] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command GetStatsVDS failed: Recovering from crash or Initializing 2017-10-01 14:54:45,804-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStatsVDSCommand] (DefaultQuartzScheduler7) [25ac73de] Command 'GetStatsVDSCommand(HostName = virt2, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vds='Host[virt2,a0ccfd11-3402-4eba-a119-4987561b738e]'})' execution failed: Recovering from crash or Initializing 2017-10-01 14:54:45,804-04 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [25ac73de] Failed getting vds stats, host='virt2'(a0ccfd11-3402-4eba-a119-4987561b738e): org.ovirt.engine.core.vdsbroker.vdsbroker.VDSRecoveringException: Recovering from crash or Initializing 2017-10-01 14:54:52,020-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler7) [1d9f9d7] Command 'GetAllVmStatsVDSCommand(HostName = virt2, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e'})' execution failed: Recovering from crash or Initializing 2017-10-01 14:54:52,020-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.PollVmStatsRefresher] (DefaultQuartzScheduler7) [1d9f9d7] Failed to fetch vms info for host 'virt2' - skipping VMs monitoring. 2017-10-01 14:54:53,172-04 ERROR [org.ovirt.vdsm.jsonrpc.client.JsonRpcClient] (ResponseWorker) [] Not able to update response for "dffd9bbf-ae4b-42db-960d-eede7b1d9a10" 2017-10-01 14:54:54,863-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [30065b39] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 14:54:59,413-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-15) [] starting processDomainRecovery for domain '63d680c7-4c94-4b29-a6cb-f288a128b9bc:iso'. 2017-10-01 14:54:59,475-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-15) [] Domain '63d680c7-4c94-4b29-a6cb-f288a128b9bc:iso' was reported by all hosts in status UP as problematic. Moving the domain to NonOperational. 2017-10-01 14:54:59,506-04 INFO [org.ovirt.engine.core.bll.storage.domain.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-6-thread-15) [66b6c976] Lock Acquired to object 'EngineLock:{exclusiveLocks='[63d680c7-4c94-4b29-a6cb-f288a128b9bc=STORAGE]', sharedLocks=''}' 2017-10-01 14:54:59,532-04 INFO [org.ovirt.engine.core.bll.storage.domain.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-6-thread-15) [66b6c976] Running command: DeactivateStorageDomainCommand internal: true. Entities affected : ID: 63d680c7-4c94-4b29-a6cb-f288a128b9bc Type: StorageAction group MANIPULATE_STORAGE_DOMAIN with role type ADMIN 2017-10-01 14:54:59,573-04 INFO [org.ovirt.engine.core.bll.storage.domain.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-6-thread-15) [66b6c976] Lock freed to object 'EngineLock:{exclusiveLocks='[63d680c7-4c94-4b29-a6cb-f288a128b9bc=STORAGE]', sharedLocks=''}' 2017-10-01 14:54:59,609-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-15) [66b6c976] EVENT_ID: SYSTEM_DEACTIVATED_STORAGE_DOMAIN(970), Correlation ID: 66b6c976, Job ID: a2983882-0428-45c1-af17-d8a2dabc26ed, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Storage Domain iso (Data Center EECS-Primary) was deactivated by system because it's not visible by any of the hosts. 2017-10-01 14:54:59,615-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-15) [66b6c976] Removing vds '[]' from the domain in maintenance cache 2017-10-01 14:54:59,615-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-15) [66b6c976] Removing host(s) '[]' from hosts unseen domain report cache 2017-10-01 14:54:59,615-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-44) [] starting processDomainRecovery for domain 'f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2:export'. 2017-10-01 14:54:59,671-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-44) [] Domain 'f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2:export' was reported by all hosts in status UP as problematic. Moving the domain to NonOperational. 2017-10-01 14:54:59,682-04 INFO [org.ovirt.engine.core.bll.storage.domain.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-6-thread-44) [3c8e26ab] Lock Acquired to object 'EngineLock:{exclusiveLocks='[f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=STORAGE]', sharedLocks=''}' 2017-10-01 14:54:59,688-04 INFO [org.ovirt.engine.core.bll.storage.domain.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-6-thread-44) [3c8e26ab] Running command: DeactivateStorageDomainCommand internal: true. Entities affected : ID: f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2 Type: StorageAction group MANIPULATE_STORAGE_DOMAIN with role type ADMIN 2017-10-01 14:54:59,691-04 INFO [org.ovirt.engine.core.bll.storage.domain.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-6-thread-44) [3c8e26ab] Lock freed to object 'EngineLock:{exclusiveLocks='[f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=STORAGE]', sharedLocks=''}' 2017-10-01 14:54:59,713-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-44) [3c8e26ab] EVENT_ID: SYSTEM_DEACTIVATED_STORAGE_DOMAIN(970), Correlation ID: 3c8e26ab, Job ID: 914421f9-81ae-4bec-a0e7-cd531c8f5ba0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Storage Domain export (Data Center EECS-Primary) was deactivated by system because it's not visible by any of the hosts. 2017-10-01 14:54:59,715-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-44) [3c8e26ab] Removing vds '[]' from the domain in maintenance cache 2017-10-01 14:54:59,715-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-44) [3c8e26ab] Removing host(s) '[]' from hosts unseen domain report cache 2017-10-01 14:55:00,032-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [662ba423] Autorecovering 1 hosts 2017-10-01 14:55:00,032-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [662ba423] Autorecovering hosts id: a0ccfd11-3402-4eba-a119-4987561b738e , name : virt2 2017-10-01 14:55:00,036-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [63eeb489] Lock Acquired to object 'EngineLock:{exclusiveLocks='[a0ccfd11-3402-4eba-a119-4987561b738e=VDS]', sharedLocks=''}' 2017-10-01 14:55:00,053-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [63eeb489] Running command: ActivateVdsCommand internal: true. Entities affected : ID: a0ccfd11-3402-4eba-a119-4987561b738e Type: VDSAction group MANIPULATE_HOST with role type ADMIN 2017-10-01 14:55:00,053-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [63eeb489] Before acquiring lock in order to prevent monitoring for host 'virt2' from data-center 'EECS-Primary' 2017-10-01 14:55:00,053-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [63eeb489] Lock acquired, from now a monitoring of host will be skipped for host 'virt2' from data-center 'EECS-Primary' 2017-10-01 14:55:00,069-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [63eeb489] START, SetVdsStatusVDSCommand(HostName = virt2, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2289287c 2017-10-01 14:55:00,069-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [63eeb489] VDS 'virt2' is spm and moved from up calling resetIrs. 2017-10-01 14:55:00,069-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (DefaultQuartzScheduler5) [63eeb489] START, ResetIrsVDSCommand( ResetIrsVDSCommandParameters:{runAsync='true', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', ignoreFailoverLimit='false', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', ignoreStopFailed='false'}), log id: 29fa2455 2017-10-01 14:55:00,084-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (DefaultQuartzScheduler5) [63eeb489] START, SpmStopVDSCommand(HostName = virt2, SpmStopVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'}), log id: 104d388f 2017-10-01 14:55:00,914-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [63eeb489] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command HSMGetAllTasksStatusesVDS failed: Recovering from crash or Initializing 2017-10-01 14:55:00,914-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler5) [63eeb489] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand' return value ' 2017-10-01 14:55:00,915-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler5) [63eeb489] HostName = virt2 2017-10-01 14:55:00,915-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler5) [63eeb489] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = virt2, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e'})' execution failed: Recovering from crash or Initializing 2017-10-01 14:55:00,915-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (DefaultQuartzScheduler5) [63eeb489] SpmStopVDSCommand::Stopping SPM on vds 'virt2', pool id 'aca61eac-ed8a-484b-bbee-3645148f6405' 2017-10-01 14:55:00,933-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [63eeb489] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command SpmStopVDS failed: Recovering from crash or Initializing 2017-10-01 14:55:00,933-04 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (DefaultQuartzScheduler5) [63eeb489] Could not stop spm of pool 'aca61eac-ed8a-484b-bbee-3645148f6405' on vds 'a0ccfd11-3402-4eba-a119-4987561b738e': Recovering from crash or Initializing 2017-10-01 14:55:00,933-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (DefaultQuartzScheduler5) [63eeb489] FINISH, SpmStopVDSCommand, log id: 104d388f 2017-10-01 14:55:00,933-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (DefaultQuartzScheduler5) [63eeb489] FINISH, ResetIrsVDSCommand, log id: 29fa2455 2017-10-01 14:55:00,935-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [63eeb489] FINISH, SetVdsStatusVDSCommand, log id: 2289287c 2017-10-01 14:55:00,935-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [63eeb489] Activate host finished. Lock released. Monitoring can run now for host 'virt2' from data-center 'EECS-Primary' 2017-10-01 14:55:00,940-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [63eeb489] EVENT_ID: VDS_ACTIVATE_FAILED_ASYNC(9,503), Correlation ID: 63eeb489, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to autorecover Host virt2. 2017-10-01 14:55:00,940-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [63eeb489] Lock freed to object 'EngineLock:{exclusiveLocks='[a0ccfd11-3402-4eba-a119-4987561b738e=VDS]', sharedLocks=''}' 2017-10-01 14:55:00,947-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [63eeb489] Autorecovering 2 storage domains 2017-10-01 14:55:00,947-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [63eeb489] Autorecovering storage domains id: 63d680c7-4c94-4b29-a6cb-f288a128b9bc 2017-10-01 14:55:00,956-04 INFO [org.ovirt.engine.core.bll.storage.connection.ConnectDomainToStorageCommand] (DefaultQuartzScheduler5) [755c787e] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 63d680c7-4c94-4b29-a6cb-f288a128b9bc Type: Storage 2017-10-01 14:55:00,956-04 INFO [org.ovirt.engine.core.bll.storage.connection.ConnectDomainToStorageCommand] (DefaultQuartzScheduler5) [755c787e] ConnectDomainToStorage. Before Connect all hosts to pool. Time: Sun Oct 01 14:55:00 EDT 2017 2017-10-01 14:55:01,039-04 INFO [org.ovirt.engine.core.bll.storage.connection.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-6-thread-24) [2052e3b5] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN 2017-10-01 14:55:01,039-04 INFO [org.ovirt.engine.core.bll.storage.connection.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-6-thread-11) [7ac79a1e] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN 2017-10-01 14:55:01,055-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-11) [7ac79a1e] START, ConnectStorageServerVDSCommand(HostName = virt1, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='00000000-0000-0000-0000-000000000000', storageType='NFS', connectionList='[StorageServerConnections:{id='13cf1e67-9961-4a25-a41f-cc76b4ab793d', connection='virtfs-fs:/nfs/iso', iqn='null', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 281b4ec0 2017-10-01 14:55:01,058-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-24) [2052e3b5] START, ConnectStorageServerVDSCommand(HostName = virt3, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='00000000-0000-0000-0000-000000000000', storageType='NFS', connectionList='[StorageServerConnections:{id='13cf1e67-9961-4a25-a41f-cc76b4ab793d', connection='virtfs-fs:/nfs/iso', iqn='null', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 60b47097 2017-10-01 14:55:01,250-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-11) [7ac79a1e] FINISH, ConnectStorageServerVDSCommand, return: {13cf1e67-9961-4a25-a41f-cc76b4ab793d=0}, log id: 281b4ec0 2017-10-01 14:55:01,260-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [78306a7] FINISH, ConnectStorageServerVDSCommand, return: {5e557659-6817-40fa-9a55-7458125404a6=0, 417fdd51-dbcd-41d9-9a05-ef4d9f485d13=0, 13cf1e67-9961-4a25-a41f-cc76b4ab793d=0}, log id: 16f9fc45 2017-10-01 14:55:01,326-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [78306a7] Host 'virt4' storage connection was succeeded 2017-10-01 14:55:01,336-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] START, ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 57bcffa5 2017-10-01 14:55:01,337-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 14:55:01,345-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-24) [2052e3b5] FINISH, ConnectStorageServerVDSCommand, return: {13cf1e67-9961-4a25-a41f-cc76b4ab793d=0}, log id: 60b47097 2017-10-01 14:55:01,348-04 INFO [org.ovirt.engine.core.bll.storage.connection.ConnectDomainToStorageCommand] (DefaultQuartzScheduler5) [755c787e] ConnectDomainToStorage. After Connect all hosts to pool. Time: Sun Oct 01 14:55:01 EDT 2017 2017-10-01 14:55:01,349-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [755c787e] Autorecovering storage domains id: f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2 2017-10-01 14:55:01,352-04 INFO [org.ovirt.engine.core.bll.storage.connection.ConnectDomainToStorageCommand] (DefaultQuartzScheduler5) [4dafcbfb] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2 Type: Storage 2017-10-01 14:55:01,353-04 INFO [org.ovirt.engine.core.bll.storage.connection.ConnectDomainToStorageCommand] (DefaultQuartzScheduler5) [4dafcbfb] ConnectDomainToStorage. Before Connect all hosts to pool. Time: Sun Oct 01 14:55:01 EDT 2017 2017-10-01 14:55:01,387-04 INFO [org.ovirt.engine.core.bll.storage.connection.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-6-thread-26) [1016960d] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN 2017-10-01 14:55:01,387-04 INFO [org.ovirt.engine.core.bll.storage.connection.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-6-thread-32) [7f01742a] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN 2017-10-01 14:55:01,402-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-26) [1016960d] START, ConnectStorageServerVDSCommand(HostName = virt1, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='00000000-0000-0000-0000-000000000000', storageType='NFS', connectionList='[StorageServerConnections:{id='417fdd51-dbcd-41d9-9a05-ef4d9f485d13', connection='virtfs-fs:/nfs/export', iqn='null', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 2c9e1ea5 2017-10-01 14:55:01,402-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-32) [7f01742a] START, ConnectStorageServerVDSCommand(HostName = virt3, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='00000000-0000-0000-0000-000000000000', storageType='NFS', connectionList='[StorageServerConnections:{id='417fdd51-dbcd-41d9-9a05-ef4d9f485d13', connection='virtfs-fs:/nfs/export', iqn='null', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 4f5a80dc 2017-10-01 14:55:01,601-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-26) [1016960d] FINISH, ConnectStorageServerVDSCommand, return: {417fdd51-dbcd-41d9-9a05-ef4d9f485d13=0}, log id: 2c9e1ea5 2017-10-01 14:55:01,700-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-40) [] FINISH, ConnectStoragePoolVDSCommand, log id: 57bcffa5 2017-10-01 14:55:01,708-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-32) [7f01742a] FINISH, ConnectStorageServerVDSCommand, return: {417fdd51-dbcd-41d9-9a05-ef4d9f485d13=0}, log id: 4f5a80dc 2017-10-01 14:55:01,710-04 INFO [org.ovirt.engine.core.bll.storage.connection.ConnectDomainToStorageCommand] (DefaultQuartzScheduler5) [4dafcbfb] ConnectDomainToStorage. After Connect all hosts to pool. Time: Sun Oct 01 14:55:01 EDT 2017 2017-10-01 14:55:02,266-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [78306a7] EVENT_ID: FENCE_OPERATION_USING_AGENT_AND_PROXY_STARTED(9,020), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Executing power management status on Host virt4 using Proxy Host virt1 and Fence Agent ipmilan:virt4-idrac. 2017-10-01 14:55:02,266-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler6) [78306a7] START, FenceVdsVDSCommand(HostName = virt1, FenceVdsVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', targetVdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', action='STATUS', agent='FenceAgent:{id='6a1fc1bb-0358-475a-aed2-e9c1868a2fc8', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', order='1', type='ipmilan', ip='virt4-idrac', port='null', user='root', password='***', encryptOptions='false', options='privlvl=OPERATOR, delay=10, lanplus=1'}', policy='null'}), log id: 5b7e05e 2017-10-01 14:55:02,411-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [3531151] FINISH, ConnectStoragePoolVDSCommand, log id: 2bcc2b36 2017-10-01 14:55:02,413-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler6) [78306a7] FINISH, FenceVdsVDSCommand, return: FenceOperationResult:{status='SUCCESS', powerStatus='ON', message=''}, log id: 5b7e05e 2017-10-01 14:55:02,428-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler6) [78306a7] START, SetMOMPolicyParametersVDSCommand(HostName = virt4, MomPolicyVDSParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846'}), log id: 175472af 2017-10-01 14:55:02,435-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler2) [3531151] hostFromVds::selectedVds - 'virt3', spmStatus 'Free', storage pool 'EECS-Primary', storage pool version '4.1' 2017-10-01 14:55:02,437-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler2) [3531151] SPM Init: could not find reported vds or not up - pool: 'EECS-Primary' vds_spm_id: '3' 2017-10-01 14:55:02,451-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler2) [3531151] SPM selection - vds seems as spm 'virt2' 2017-10-01 14:55:02,466-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (DefaultQuartzScheduler2) [3531151] START, SpmStopVDSCommand(HostName = virt2, SpmStopVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'}), log id: 3c09502f 2017-10-01 14:55:02,473-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3531151] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command HSMGetAllTasksStatusesVDS failed: Recovering from crash or Initializing 2017-10-01 14:55:02,473-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler2) [3531151] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand' return value ' 2017-10-01 14:55:02,473-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler2) [3531151] HostName = virt2 2017-10-01 14:55:02,473-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler2) [3531151] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = virt2, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e'})' execution failed: Recovering from crash or Initializing 2017-10-01 14:55:02,473-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (DefaultQuartzScheduler2) [3531151] SpmStopVDSCommand::Stopping SPM on vds 'virt2', pool id 'aca61eac-ed8a-484b-bbee-3645148f6405' 2017-10-01 14:55:02,473-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler6) [78306a7] FINISH, SetMOMPolicyParametersVDSCommand, log id: 175472af 2017-10-01 14:55:02,492-04 INFO [org.ovirt.engine.core.bll.hostdev.RefreshHostDevicesCommand] (DefaultQuartzScheduler6) [2cc538a8] Running command: RefreshHostDevicesCommand internal: true. Entities affected : ID: 5ae29ed2-9c0f-4754-8df3-b43d14726846 Type: VDSAction group MANIPULATE_HOST with role type ADMIN 2017-10-01 14:55:02,497-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3531151] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command SpmStopVDS failed: Recovering from crash or Initializing 2017-10-01 14:55:02,497-04 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (DefaultQuartzScheduler2) [3531151] Could not stop spm of pool 'aca61eac-ed8a-484b-bbee-3645148f6405' on vds 'a0ccfd11-3402-4eba-a119-4987561b738e': Recovering from crash or Initializing 2017-10-01 14:55:02,497-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (DefaultQuartzScheduler2) [3531151] FINISH, SpmStopVDSCommand, log id: 3c09502f 2017-10-01 14:55:02,497-04 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler2) [3531151] spm stop on spm failed, stopping spm selection! 2017-10-01 14:55:02,744-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [2cc538a8] EVENT_ID: VDS_DETECTED(13), Correlation ID: 3eabca18, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Status of host virt4 was set to Up. 2017-10-01 14:55:02,746-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [2cc538a8] EVENT_ID: VDS_FENCE_STATUS(496), Correlation ID: 3eabca18, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host virt4 power management was verified successfully. 2017-10-01 14:55:02,779-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [738f9d5f] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5ae29ed2-9c0f-4754-8df3-b43d14726846 Type: VDS 2017-10-01 14:55:02,783-04 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler6) [738f9d5f] Storage Domain 'f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2:export' was reported by Host 'virt4' as Active in Pool 'aca61eac-ed8a-484b-bbee-3645148f6405', moving to active status 2017-10-01 14:55:02,793-04 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler6) [738f9d5f] Storage Domain '63d680c7-4c94-4b29-a6cb-f288a128b9bc:iso' was reported by Host 'virt4' as Active in Pool 'aca61eac-ed8a-484b-bbee-3645148f6405', moving to active status 2017-10-01 14:55:02,800-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-29) [] Domain 'c8653dba-63ec-4447-872c-0ffcc6af9c0f:data' recovered from problem. vds: 'virt4' 2017-10-01 14:55:05,881-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [7b8249be] START, GetHardwareInfoVDSCommand(HostName = virt2, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vds='Host[virt2,a0ccfd11-3402-4eba-a119-4987561b738e]'}), log id: 32ddbbe7 2017-10-01 14:55:06,887-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [7b8249be] FINISH, GetHardwareInfoVDSCommand, log id: 32ddbbe7 2017-10-01 14:55:06,981-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [38af2d3a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: a0ccfd11-3402-4eba-a119-4987561b738e Type: VDS 2017-10-01 14:55:07,007-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [17bd923e] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: aca61eac-ed8a-484b-bbee-3645148f6405 Type: StoragePool 2017-10-01 14:55:07,014-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [35c57e9d] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: aca61eac-ed8a-484b-bbee-3645148f6405 Type: StoragePool 2017-10-01 14:55:07,027-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [35c57e9d] START, ConnectStorageServerVDSCommand(HostName = virt2, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', storageType='NFS', connectionList='[StorageServerConnections:{id='5e557659-6817-40fa-9a55-7458125404a6', connection='virtfs-fs:/nfs/data', iqn='null', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='13cf1e67-9961-4a25-a41f-cc76b4ab793d', connection='virtfs-fs:/nfs/iso', iqn='null', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='417fdd51-dbcd-41d9-9a05-ef4d9f485d13', connection='virtfs-fs:/nfs/export', iqn='null', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 7e4d18f6 2017-10-01 14:55:07,030-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler5) [6debc027] VM 'a82b4cc3-6ef6-4a8e-97ef-1d4ea1b7e07c'(indigo) moved from 'NotResponding' --> 'Up' 2017-10-01 14:55:07,030-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler5) [6debc027] VM '3bb83ad5-dc00-445d-8708-2db18c53e0e6'(www-vhost) moved from 'NotResponding' --> 'Up' 2017-10-01 14:55:07,030-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler5) [6debc027] VM 'a8d4e149-55ec-4961-a414-d7d1e1b05a59'(fix) moved from 'NotResponding' --> 'Up' 2017-10-01 14:55:07,051-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [35c57e9d] FINISH, ConnectStorageServerVDSCommand, return: {5e557659-6817-40fa-9a55-7458125404a6=0, 417fdd51-dbcd-41d9-9a05-ef4d9f485d13=0, 13cf1e67-9961-4a25-a41f-cc76b4ab793d=0}, log id: 7e4d18f6 2017-10-01 14:55:07,055-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [35c57e9d] Host 'virt2' storage connection was succeeded 2017-10-01 14:55:07,065-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-10) [] START, ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 5ed53e46 2017-10-01 14:55:07,065-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-10) [] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 14:55:07,067-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler10) [46534527] VM 'bcd5776e-978b-4723-933d-70a80322a796'(cape) moved from 'NotResponding' --> 'Up' 2017-10-01 14:55:07,067-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler10) [46534527] VM '3e5a5728-8940-4fe7-b4af-11fb8e12f116'(www) moved from 'NotResponding' --> 'Up' 2017-10-01 14:55:07,067-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler10) [46534527] VM 'f87ec08d-0fdf-42ba-8aac-c66c404d7aae'(webapp) moved from 'NotResponding' --> 'Up' 2017-10-01 14:55:07,067-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler10) [46534527] VM 'd66235b4-7c8d-4da2-a572-82ecc9c53a59'(www-las) moved from 'NotResponding' --> 'Up' 2017-10-01 14:55:07,067-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler10) [46534527] VM '5885c43a-b133-45dc-b8d5-1aa2a4a19579'(prs) moved from 'NotResponding' --> 'Up' 2017-10-01 14:55:07,067-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler10) [46534527] VM 'ae65a104-ed3f-40e0-ba92-5708cf338918'(vpn2) moved from 'NotResponding' --> 'Up' 2017-10-01 14:55:07,067-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler10) [46534527] VM 'd56363ad-9cb1-4a70-a33a-ce439b044d7d'(svn) moved from 'NotResponding' --> 'Up' 2017-10-01 14:55:07,075-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler1) [5c961d92] VM '1c7e966f-29e8-4788-b64c-86e52f415061'(udb) moved from 'NotResponding' --> 'Up' 2017-10-01 14:55:07,075-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler1) [5c961d92] VM '9ea2821e-557f-45cb-9872-1e6c5aeaf246'(rs) moved from 'NotResponding' --> 'Up' 2017-10-01 14:55:08,251-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-10) [] FINISH, ConnectStoragePoolVDSCommand, log id: 5ed53e46 2017-10-01 14:55:09,265-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-46) [] starting processDomainRecovery for domain 'c8653dba-63ec-4447-872c-0ffcc6af9c0f:data'. 2017-10-01 14:55:09,321-04 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-46) [] vds 'virt3' reported domain 'c8653dba-63ec-4447-872c-0ffcc6af9c0f:data' as in problem, attempting to move the vds to status NonOperational 2017-10-01 14:55:09,321-04 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-46) [] vds 'virt1' reported domain 'c8653dba-63ec-4447-872c-0ffcc6af9c0f:data' as in problem, attempting to move the vds to status NonOperational 2017-10-01 14:55:09,321-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-46) [] Removing vds '[e1e71e9d-a0cd-47db-96c6-f724f6cea736, 45f8b331-842e-48e7-9df8-56adddb93836]' from the domain in maintenance cache 2017-10-01 14:55:09,321-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-46) [] Removing host(s) '[e1e71e9d-a0cd-47db-96c6-f724f6cea736, 45f8b331-842e-48e7-9df8-56adddb93836]' from hosts unseen domain report cache 2017-10-01 14:55:09,349-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (org.ovirt.thread.pool-6-thread-39) [76e2b257] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: e1e71e9d-a0cd-47db-96c6-f724f6cea736 Type: VDS 2017-10-01 14:55:09,365-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-6-thread-39) [76e2b257] START, SetVdsStatusVDSCommand(HostName = virt3, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4fb00079 2017-10-01 14:55:09,371-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (org.ovirt.thread.pool-6-thread-1) [5ed5262e] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 45f8b331-842e-48e7-9df8-56adddb93836 Type: VDS 2017-10-01 14:55:09,386-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-6-thread-1) [5ed5262e] START, SetVdsStatusVDSCommand(HostName = virt1, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: fd9998c 2017-10-01 14:55:09,388-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-6-thread-1) [5ed5262e] FINISH, SetVdsStatusVDSCommand, log id: fd9998c 2017-10-01 14:55:09,427-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler6) [16b7e376] VM 'db3e08fb-c111-442d-aeec-1507da30f486'(print) moved from 'NotResponding' --> 'Up' 2017-10-01 14:55:09,427-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler6) [16b7e376] VM '911d6883-607c-464e-b46b-8e6141bdd2de'(hdb) moved from 'NotResponding' --> 'Up' 2017-10-01 14:55:09,427-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler6) [16b7e376] VM 'ddf37076-e795-4c87-a4fb-9efcaae76d72'(dc1) moved from 'NotResponding' --> 'Up' 2017-10-01 14:55:09,427-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler6) [16b7e376] VM 'bfe9a524-2238-4bab-a91f-5dc6ed270896'(pcmode) moved from 'NotResponding' --> 'Up' 2017-10-01 14:55:09,427-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler6) [16b7e376] VM '1325cebf-834e-4571-a3e1-82beb0021ffe'(dns1) moved from 'NotResponding' --> 'Up' 2017-10-01 14:55:09,427-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler6) [16b7e376] VM '8391f9a7-7fe7-4025-8fba-6273ae587a52'(archive) moved from 'NotResponding' --> 'Up' 2017-10-01 14:55:09,428-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler6) [16b7e376] VM 'c8682859-9e55-460a-9ef0-d03e4a85f50e'(dhcp1) moved from 'NotResponding' --> 'Up' 2017-10-01 14:55:09,428-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler6) [16b7e376] VM 'aa487207-7ff4-465a-9d9b-2a103d50dc77'(ftp1) moved from 'NotResponding' --> 'Up' 2017-10-01 14:55:09,428-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [35c57e9d] EVENT_ID: FENCE_OPERATION_USING_AGENT_AND_PROXY_STARTED(9,020), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Executing power management status on Host virt2 using Proxy Host virt3 and Fence Agent ipmilan:virt2-idrac. 2017-10-01 14:55:09,428-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler2) [35c57e9d] START, FenceVdsVDSCommand(HostName = virt3, FenceVdsVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', targetVdsId='a0ccfd11-3402-4eba-a119-4987561b738e', action='STATUS', agent='FenceAgent:{id='7c392b88-1692-461f-9248-12f13b0e0b01', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', order='1', type='ipmilan', ip='virt2-idrac', port='null', user='root', password='***', encryptOptions='false', options='privlvl=OPERATOR, delay=10, lanplus=1'}', policy='null'}), log id: 51c02005 2017-10-01 14:55:09,435-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-6-thread-39) [76e2b257] FINISH, SetVdsStatusVDSCommand, log id: 4fb00079 2017-10-01 14:55:09,444-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-1) [5ed5262e] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 5ed5262e, Job ID: f27a17ef-aa93-4252-9e03-7992c8afaf76, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host virt1 cannot access the Storage Domain(s) data attached to the Data Center EECS-Primary. Setting Host state to Non-Operational. 2017-10-01 14:55:09,449-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-39) [76e2b257] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 76e2b257, Job ID: 326bf6bc-461b-4dd9-8cde-2ab8cb01711c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host virt3 cannot access the Storage Domain(s) data attached to the Data Center EECS-Primary. Setting Host state to Non-Operational. 2017-10-01 14:55:09,500-04 INFO [org.ovirt.engine.core.bll.MigrateVmCommand] (org.ovirt.thread.pool-6-thread-47) [31042548] Lock Acquired to object 'EngineLock:{exclusiveLocks='[899e3a1c-dcc4-4426-9dd4-0f1e8b94a2b8=VM]', sharedLocks=''}' 2017-10-01 14:55:09,500-04 INFO [org.ovirt.engine.core.bll.MigrateVmCommand] (org.ovirt.thread.pool-6-thread-42) [58940caa] Lock Acquired to object 'EngineLock:{exclusiveLocks='[1c7e966f-29e8-4788-b64c-86e52f415061=VM]', sharedLocks=''}' 2017-10-01 14:55:09,517-04 WARN [org.ovirt.engine.core.bll.MigrateVmCommand] (org.ovirt.thread.pool-6-thread-47) [31042548] Validation of action 'MigrateVm' failed for user SYSTEM. Reasons: VAR__ACTION__MIGRATE,VAR__TYPE__VM,ACTION_TYPE_FAILED_VM_STATUS_ILLEGAL,VAR__VM_STATUS__NOT_RESPONDING 2017-10-01 14:55:09,518-04 INFO [org.ovirt.engine.core.bll.MigrateVmCommand] (org.ovirt.thread.pool-6-thread-47) [31042548] Lock freed to object 'EngineLock:{exclusiveLocks='[899e3a1c-dcc4-4426-9dd4-0f1e8b94a2b8=VM]', sharedLocks=''}' 2017-10-01 14:55:09,521-04 ERROR [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (org.ovirt.thread.pool-6-thread-47) [31042548] Failed to migrate VM 'ipa' 2017-10-01 14:55:09,544-04 INFO [org.ovirt.engine.core.bll.MigrateVmCommand] (org.ovirt.thread.pool-6-thread-47) [4e6b2aa0] Lock Acquired to object 'EngineLock:{exclusiveLocks='[911d6883-607c-464e-b46b-8e6141bdd2de=VM]', sharedLocks=''}' 2017-10-01 14:55:09,637-04 INFO [org.ovirt.engine.core.bll.MigrateVmCommand] (org.ovirt.thread.pool-6-thread-42) [58940caa] Running command: MigrateVmCommand internal: true. Entities affected : ID: 1c7e966f-29e8-4788-b64c-86e52f415061 Type: VMAction group MIGRATE_VM with role type USER 2017-10-01 14:55:09,661-04 INFO [org.ovirt.engine.core.bll.MigrateVmCommand] (org.ovirt.thread.pool-6-thread-47) [4e6b2aa0] Running command: MigrateVmCommand internal: true. Entities affected : ID: 911d6883-607c-464e-b46b-8e6141bdd2de Type: VMAction group MIGRATE_VM with role type USER 2017-10-01 14:55:09,724-04 INFO [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand] (org.ovirt.thread.pool-6-thread-42) [58940caa] START, MigrateVDSCommand( MigrateVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vmId='1c7e966f-29e8-4788-b64c-86e52f415061', srcHost='virt3-mgmt', dstVdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', dstHost='virt4-mgmt:54321', migrationMethod='ONLINE', tunnelMigration='false', migrationDowntime='0', autoConverge='true', migrateCompressed='false', consoleAddress='null', maxBandwidth='null', enableGuestEvents='true', maxIncomingMigrations='2', maxOutgoingMigrations='2', convergenceSchedule='[init=[{name=setDowntime, params=[100]}], stalling=[{limit=1, action={name=setDowntime, params=[150]}}, {limit=2, action={name=setDowntime, params=[200]}}, {limit=3, action={name=setDowntime, params=[300]}}, {limit=4, action={name=setDowntime, params=[400]}}, {limit=6, action={name=setDowntime, params=[500]}}, {limit=-1, action={name=abort, params=[]}}]]'}), log id: 76583b66 2017-10-01 14:55:09,724-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] (org.ovirt.thread.pool-6-thread-42) [58940caa] START, MigrateBrokerVDSCommand(HostName = virt3, MigrateVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vmId='1c7e966f-29e8-4788-b64c-86e52f415061', srcHost='virt3-mgmt', dstVdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', dstHost='virt4-mgmt:54321', migrationMethod='ONLINE', tunnelMigration='false', migrationDowntime='0', autoConverge='true', migrateCompressed='false', consoleAddress='null', maxBandwidth='null', enableGuestEvents='true', maxIncomingMigrations='2', maxOutgoingMigrations='2', convergenceSchedule='[init=[{name=setDowntime, params=[100]}], stalling=[{limit=1, action={name=setDowntime, params=[150]}}, {limit=2, action={name=setDowntime, params=[200]}}, {limit=3, action={name=setDowntime, params=[300]}}, {limit=4, action={name=setDowntime, params=[400]}}, {limit=6, action={name=setDowntime, params=[500]}}, {limit=-1, action={name=abort, params=[]}}]]'}), log id: 5a948feb 2017-10-01 14:55:09,732-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler2) [35c57e9d] FINISH, FenceVdsVDSCommand, return: FenceOperationResult:{status='SUCCESS', powerStatus='ON', message=''}, log id: 51c02005 2017-10-01 14:55:09,748-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler2) [35c57e9d] START, SetMOMPolicyParametersVDSCommand(HostName = virt2, MomPolicyVDSParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e'}), log id: 30738f3a 2017-10-01 14:55:09,752-04 INFO [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand] (org.ovirt.thread.pool-6-thread-47) [4e6b2aa0] START, MigrateVDSCommand( MigrateVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vmId='911d6883-607c-464e-b46b-8e6141bdd2de', srcHost='virt1-mgmt', dstVdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', dstHost='virt4-mgmt:54321', migrationMethod='ONLINE', tunnelMigration='false', migrationDowntime='0', autoConverge='true', migrateCompressed='false', consoleAddress='null', maxBandwidth='null', enableGuestEvents='true', maxIncomingMigrations='2', maxOutgoingMigrations='2', convergenceSchedule='[init=[{name=setDowntime, params=[100]}], stalling=[{limit=1, action={name=setDowntime, params=[150]}}, {limit=2, action={name=setDowntime, params=[200]}}, {limit=3, action={name=setDowntime, params=[300]}}, {limit=4, action={name=setDowntime, params=[400]}}, {limit=6, action={name=setDowntime, params=[500]}}, {limit=-1, action={name=abort, params=[]}}]]'}), log id: 17718d96 2017-10-01 14:55:09,753-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] (org.ovirt.thread.pool-6-thread-47) [4e6b2aa0] START, MigrateBrokerVDSCommand(HostName = virt1, MigrateVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vmId='911d6883-607c-464e-b46b-8e6141bdd2de', srcHost='virt1-mgmt', dstVdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', dstHost='virt4-mgmt:54321', migrationMethod='ONLINE', tunnelMigration='false', migrationDowntime='0', autoConverge='true', migrateCompressed='false', consoleAddress='null', maxBandwidth='null', enableGuestEvents='true', maxIncomingMigrations='2', maxOutgoingMigrations='2', convergenceSchedule='[init=[{name=setDowntime, params=[100]}], stalling=[{limit=1, action={name=setDowntime, params=[150]}}, {limit=2, action={name=setDowntime, params=[200]}}, {limit=3, action={name=setDowntime, params=[300]}}, {limit=4, action={name=setDowntime, params=[400]}}, {limit=6, action={name=setDowntime, params=[500]}}, {limit=-1, action={name=abort, params=[]}}]]'}), log id: 7142b591 2017-10-01 14:55:09,767-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] (org.ovirt.thread.pool-6-thread-42) [58940caa] FINISH, MigrateBrokerVDSCommand, log id: 5a948feb 2017-10-01 14:55:09,769-04 INFO [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand] (org.ovirt.thread.pool-6-thread-42) [58940caa] FINISH, MigrateVDSCommand, return: MigratingFrom, log id: 76583b66 2017-10-01 14:55:09,774-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-42) [58940caa] EVENT_ID: VM_MIGRATION_START_SYSTEM_INITIATED(67), Correlation ID: 58940caa, Job ID: 37f56cd4-078d-4bec-bce5-ab1ca2c5cda2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Migration initiated by system (VM: udb, Source: virt3, Destination: virt4, Reason: Host preparing for maintenance). 2017-10-01 14:55:09,785-04 INFO [org.ovirt.engine.core.bll.MigrateVmCommand] (org.ovirt.thread.pool-6-thread-42) [49a0af35] Lock Acquired to object 'EngineLock:{exclusiveLocks='[64734fca-4d6c-443c-82d1-abf7cf106601=VM]', sharedLocks=''}' 2017-10-01 14:55:09,801-04 WARN [org.ovirt.engine.core.bll.MigrateVmCommand] (org.ovirt.thread.pool-6-thread-42) [49a0af35] Validation of action 'MigrateVm' failed for user SYSTEM. Reasons: VAR__ACTION__MIGRATE,VAR__TYPE__VM,ACTION_TYPE_FAILED_VM_STATUS_ILLEGAL,VAR__VM_STATUS__NOT_RESPONDING 2017-10-01 14:55:09,802-04 INFO [org.ovirt.engine.core.bll.MigrateVmCommand] (org.ovirt.thread.pool-6-thread-42) [49a0af35] Lock freed to object 'EngineLock:{exclusiveLocks='[64734fca-4d6c-443c-82d1-abf7cf106601=VM]', sharedLocks=''}' 2017-10-01 14:55:09,805-04 ERROR [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (org.ovirt.thread.pool-6-thread-42) [49a0af35] Failed to migrate VM 'webdav' 2017-10-01 14:55:09,816-04 INFO [org.ovirt.engine.core.bll.MigrateVmCommand] (org.ovirt.thread.pool-6-thread-42) [c313d7f] Lock Acquired to object 'EngineLock:{exclusiveLocks='[cc1dbe93-bb80-451e-8b3b-c9cc5e90432f=VM]', sharedLocks=''}' 2017-10-01 14:55:09,831-04 WARN [org.ovirt.engine.core.bll.MigrateVmCommand] (org.ovirt.thread.pool-6-thread-42) [c313d7f] Validation of action 'MigrateVm' failed for user SYSTEM. Reasons: VAR__ACTION__MIGRATE,VAR__TYPE__VM,ACTION_TYPE_FAILED_VM_STATUS_ILLEGAL,VAR__VM_STATUS__NOT_RESPONDING 2017-10-01 14:55:09,832-04 INFO [org.ovirt.engine.core.bll.MigrateVmCommand] (org.ovirt.thread.pool-6-thread-42) [c313d7f] Lock freed to object 'EngineLock:{exclusiveLocks='[cc1dbe93-bb80-451e-8b3b-c9cc5e90432f=VM]', sharedLocks=''}' 2017-10-01 14:55:09,836-04 ERROR [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (org.ovirt.thread.pool-6-thread-42) [c313d7f] Failed to migrate VM 'license' 2017-10-01 14:55:09,846-04 INFO [org.ovirt.engine.core.bll.MigrateVmCommand] (org.ovirt.thread.pool-6-thread-42) [1a031d8d] Lock Acquired to object 'EngineLock:{exclusiveLocks='[9ea2821e-557f-45cb-9872-1e6c5aeaf246=VM]', sharedLocks=''}' 2017-10-01 14:55:09,906-04 INFO [org.ovirt.engine.core.bll.MigrateVmCommand] (org.ovirt.thread.pool-6-thread-42) [1a031d8d] Running command: MigrateVmCommand internal: true. Entities affected : ID: 9ea2821e-557f-45cb-9872-1e6c5aeaf246 Type: VMAction group MIGRATE_VM with role type USER 2017-10-01 14:55:09,971-04 INFO [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand] (org.ovirt.thread.pool-6-thread-42) [1a031d8d] START, MigrateVDSCommand( MigrateVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vmId='9ea2821e-557f-45cb-9872-1e6c5aeaf246', srcHost='virt3-mgmt', dstVdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', dstHost='virt4-mgmt:54321', migrationMethod='ONLINE', tunnelMigration='false', migrationDowntime='0', autoConverge='true', migrateCompressed='false', consoleAddress='null', maxBandwidth='null', enableGuestEvents='true', maxIncomingMigrations='2', maxOutgoingMigrations='2', convergenceSchedule='[init=[{name=setDowntime, params=[100]}], stalling=[{limit=1, action={name=setDowntime, params=[150]}}, {limit=2, action={name=setDowntime, params=[200]}}, {limit=3, action={name=setDowntime, params=[300]}}, {limit=4, action={name=setDowntime, params=[400]}}, {limit=6, action={name=setDowntime, params=[500]}}, {limit=-1, action={name=abort, params=[]}}]]'}), log id: 2fc1acee 2017-10-01 14:55:09,972-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] (org.ovirt.thread.pool-6-thread-42) [1a031d8d] START, MigrateBrokerVDSCommand(HostName = virt3, MigrateVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vmId='9ea2821e-557f-45cb-9872-1e6c5aeaf246', srcHost='virt3-mgmt', dstVdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', dstHost='virt4-mgmt:54321', migrationMethod='ONLINE', tunnelMigration='false', migrationDowntime='0', autoConverge='true', migrateCompressed='false', consoleAddress='null', maxBandwidth='null', enableGuestEvents='true', maxIncomingMigrations='2', maxOutgoingMigrations='2', convergenceSchedule='[init=[{name=setDowntime, params=[100]}], stalling=[{limit=1, action={name=setDowntime, params=[150]}}, {limit=2, action={name=setDowntime, params=[200]}}, {limit=3, action={name=setDowntime, params=[300]}}, {limit=4, action={name=setDowntime, params=[400]}}, {limit=6, action={name=setDowntime, params=[500]}}, {limit=-1, action={name=abort, params=[]}}]]'}), log id: 7f6eba4b 2017-10-01 14:55:09,973-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler2) [35c57e9d] FINISH, SetMOMPolicyParametersVDSCommand, log id: 30738f3a 2017-10-01 14:55:09,977-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] (org.ovirt.thread.pool-6-thread-47) [4e6b2aa0] FINISH, MigrateBrokerVDSCommand, log id: 7142b591 2017-10-01 14:55:09,979-04 INFO [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand] (org.ovirt.thread.pool-6-thread-47) [4e6b2aa0] FINISH, MigrateVDSCommand, return: MigratingFrom, log id: 17718d96 2017-10-01 14:55:09,994-04 INFO [org.ovirt.engine.core.bll.hostdev.RefreshHostDevicesCommand] (DefaultQuartzScheduler2) [37b78e0f] Running command: RefreshHostDevicesCommand internal: true. Entities affected : ID: a0ccfd11-3402-4eba-a119-4987561b738e Type: VDSAction group MANIPULATE_HOST with role type ADMIN 2017-10-01 14:55:09,998-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-47) [4e6b2aa0] EVENT_ID: VM_MIGRATION_START_SYSTEM_INITIATED(67), Correlation ID: 4e6b2aa0, Job ID: b2979192-145f-4a51-a56d-7bdefc27d56a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Migration initiated by system (VM: hdb, Source: virt1, Destination: virt4, Reason: Host preparing for maintenance). 2017-10-01 14:55:10,016-04 INFO [org.ovirt.engine.core.bll.MigrateVmCommand] (org.ovirt.thread.pool-6-thread-47) [346380fe] Lock Acquired to object 'EngineLock:{exclusiveLocks='[77766cb4-0625-4432-a16f-def5e702102a=VM]', sharedLocks=''}' 2017-10-01 14:55:10,031-04 WARN [org.ovirt.engine.core.bll.MigrateVmCommand] (org.ovirt.thread.pool-6-thread-47) [346380fe] Validation of action 'MigrateVm' failed for user SYSTEM. Reasons: VAR__ACTION__MIGRATE,VAR__TYPE__VM,ACTION_TYPE_FAILED_VM_STATUS_ILLEGAL,VAR__VM_STATUS__NOT_RESPONDING 2017-10-01 14:55:10,032-04 INFO [org.ovirt.engine.core.bll.MigrateVmCommand] (org.ovirt.thread.pool-6-thread-47) [346380fe] Lock freed to object 'EngineLock:{exclusiveLocks='[77766cb4-0625-4432-a16f-def5e702102a=VM]', sharedLocks=''}' 2017-10-01 14:55:10,036-04 ERROR [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (org.ovirt.thread.pool-6-thread-47) [346380fe] Failed to migrate VM 'labtest1' 2017-10-01 14:55:10,046-04 INFO [org.ovirt.engine.core.bll.MigrateVmCommand] (org.ovirt.thread.pool-6-thread-47) [2d950fdc] Lock Acquired to object 'EngineLock:{exclusiveLocks='[8391f9a7-7fe7-4025-8fba-6273ae587a52=VM]', sharedLocks=''}' 2017-10-01 14:55:10,108-04 INFO [org.ovirt.engine.core.bll.MigrateVmCommand] (org.ovirt.thread.pool-6-thread-47) [2d950fdc] Running command: MigrateVmCommand internal: true. Entities affected : ID: 8391f9a7-7fe7-4025-8fba-6273ae587a52 Type: VMAction group MIGRATE_VM with role type USER 2017-10-01 14:55:10,121-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] (org.ovirt.thread.pool-6-thread-42) [1a031d8d] FINISH, MigrateBrokerVDSCommand, log id: 7f6eba4b 2017-10-01 14:55:10,123-04 INFO [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand] (org.ovirt.thread.pool-6-thread-42) [1a031d8d] FINISH, MigrateVDSCommand, return: MigratingFrom, log id: 2fc1acee 2017-10-01 14:55:10,127-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-42) [1a031d8d] EVENT_ID: VM_MIGRATION_START_SYSTEM_INITIATED(67), Correlation ID: 1a031d8d, Job ID: 1d6ef5c3-aab0-40f3-97f1-52aee6279df1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Migration initiated by system (VM: rs, Source: virt3, Destination: virt4, Reason: Host preparing for maintenance). 2017-10-01 14:55:10,140-04 INFO [org.ovirt.engine.core.bll.MigrateVmCommand] (org.ovirt.thread.pool-6-thread-42) [57b57c8c] Lock Acquired to object 'EngineLock:{exclusiveLocks='[1734fed5-fc97-4714-bb63-55e8f4eb6d34=VM]', sharedLocks=''}' 2017-10-01 14:55:10,156-04 WARN [org.ovirt.engine.core.bll.MigrateVmCommand] (org.ovirt.thread.pool-6-thread-42) [57b57c8c] Validation of action 'MigrateVm' failed for user SYSTEM. Reasons: VAR__ACTION__MIGRATE,VAR__TYPE__VM,ACTION_TYPE_FAILED_VM_STATUS_ILLEGAL,VAR__VM_STATUS__NOT_RESPONDING 2017-10-01 14:55:10,156-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [37b78e0f] EVENT_ID: VDS_DETECTED(13), Correlation ID: 17bd923e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Status of host virt2 was set to Up. 2017-10-01 14:55:10,157-04 INFO [org.ovirt.engine.core.bll.MigrateVmCommand] (org.ovirt.thread.pool-6-thread-42) [57b57c8c] Lock freed to object 'EngineLock:{exclusiveLocks='[1734fed5-fc97-4714-bb63-55e8f4eb6d34=VM]', sharedLocks=''}' 2017-10-01 14:55:10,159-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [37b78e0f] EVENT_ID: VDS_FENCE_STATUS(496), Correlation ID: 17bd923e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host virt2 power management was verified successfully. 2017-10-01 14:55:10,160-04 ERROR [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (org.ovirt.thread.pool-6-thread-42) [57b57c8c] Failed to migrate VM 'wiki' 2017-10-01 14:55:10,181-04 INFO [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand] (org.ovirt.thread.pool-6-thread-47) [2d950fdc] START, MigrateVDSCommand( MigrateVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vmId='8391f9a7-7fe7-4025-8fba-6273ae587a52', srcHost='virt1-mgmt', dstVdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', dstHost='virt4-mgmt:54321', migrationMethod='ONLINE', tunnelMigration='false', migrationDowntime='0', autoConverge='true', migrateCompressed='false', consoleAddress='null', maxBandwidth='null', enableGuestEvents='true', maxIncomingMigrations='2', maxOutgoingMigrations='2', convergenceSchedule='[init=[{name=setDowntime, params=[100]}], stalling=[{limit=1, action={name=setDowntime, params=[150]}}, {limit=2, action={name=setDowntime, params=[200]}}, {limit=3, action={name=setDowntime, params=[300]}}, {limit=4, action={name=setDowntime, params=[400]}}, {limit=6, action={name=setDowntime, params=[500]}}, {limit=-1, action={name=abort, params=[]}}]]'}), log id: 37c5a4d 2017-10-01 14:55:10,182-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] (org.ovirt.thread.pool-6-thread-47) [2d950fdc] START, MigrateBrokerVDSCommand(HostName = virt1, MigrateVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vmId='8391f9a7-7fe7-4025-8fba-6273ae587a52', srcHost='virt1-mgmt', dstVdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', dstHost='virt4-mgmt:54321', migrationMethod='ONLINE', tunnelMigration='false', migrationDowntime='0', autoConverge='true', migrateCompressed='false', consoleAddress='null', maxBandwidth='null', enableGuestEvents='true', maxIncomingMigrations='2', maxOutgoingMigrations='2', convergenceSchedule='[init=[{name=setDowntime, params=[100]}], stalling=[{limit=1, action={name=setDowntime, params=[150]}}, {limit=2, action={name=setDowntime, params=[200]}}, {limit=3, action={name=setDowntime, params=[300]}}, {limit=4, action={name=setDowntime, params=[400]}}, {limit=6, action={name=setDowntime, params=[500]}}, {limit=-1, action={name=abort, params=[]}}]]'}), log id: 15bdbf6e 2017-10-01 14:55:10,184-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [3c7f52de] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: a0ccfd11-3402-4eba-a119-4987561b738e Type: VDS 2017-10-01 14:55:10,220-04 INFO [org.ovirt.engine.core.bll.MigrateVmCommand] (org.ovirt.thread.pool-6-thread-42) [35b8a8f0] Lock Acquired to object 'EngineLock:{exclusiveLocks='[c64d80f9-a3bd-44f3-89c4-f1b9b509cdde=VM]', sharedLocks=''}' 2017-10-01 14:55:10,236-04 WARN [org.ovirt.engine.core.bll.MigrateVmCommand] (org.ovirt.thread.pool-6-thread-42) [35b8a8f0] Validation of action 'MigrateVm' failed for user SYSTEM. Reasons: VAR__ACTION__MIGRATE,VAR__TYPE__VM,ACTION_TYPE_FAILED_VM_STATUS_ILLEGAL,VAR__VM_STATUS__NOT_RESPONDING 2017-10-01 14:55:10,236-04 INFO [org.ovirt.engine.core.bll.MigrateVmCommand] (org.ovirt.thread.pool-6-thread-42) [35b8a8f0] Lock freed to object 'EngineLock:{exclusiveLocks='[c64d80f9-a3bd-44f3-89c4-f1b9b509cdde=VM]', sharedLocks=''}' 2017-10-01 14:55:10,262-04 ERROR [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (org.ovirt.thread.pool-6-thread-42) [35b8a8f0] Failed to migrate VM 'red' 2017-10-01 14:55:12,190-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] (org.ovirt.thread.pool-6-thread-47) [2d950fdc] FINISH, MigrateBrokerVDSCommand, log id: 15bdbf6e 2017-10-01 14:55:12,193-04 INFO [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand] (org.ovirt.thread.pool-6-thread-47) [2d950fdc] FINISH, MigrateVDSCommand, return: MigratingFrom, log id: 37c5a4d 2017-10-01 14:55:12,197-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-47) [2d950fdc] EVENT_ID: VM_MIGRATION_START_SYSTEM_INITIATED(67), Correlation ID: 2d950fdc, Job ID: 332db2e9-620c-4c11-a225-018a7ad89be1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Migration initiated by system (VM: archive, Source: virt1, Destination: virt4, Reason: Host preparing for maintenance). 2017-10-01 14:55:12,216-04 INFO [org.ovirt.engine.core.bll.MigrateVmCommand] (org.ovirt.thread.pool-6-thread-47) [267b35df] Lock Acquired to object 'EngineLock:{exclusiveLocks='[59287796-c634-4f3c-94b1-2c9699986637=VM]', sharedLocks=''}' 2017-10-01 14:55:12,232-04 WARN [org.ovirt.engine.core.bll.MigrateVmCommand] (org.ovirt.thread.pool-6-thread-47) [267b35df] Validation of action 'MigrateVm' failed for user SYSTEM. Reasons: VAR__ACTION__MIGRATE,VAR__TYPE__VM,ACTION_TYPE_FAILED_VM_STATUS_ILLEGAL,VAR__VM_STATUS__NOT_RESPONDING 2017-10-01 14:55:12,232-04 INFO [org.ovirt.engine.core.bll.MigrateVmCommand] (org.ovirt.thread.pool-6-thread-47) [267b35df] Lock freed to object 'EngineLock:{exclusiveLocks='[59287796-c634-4f3c-94b1-2c9699986637=VM]', sharedLocks=''}' 2017-10-01 14:55:12,236-04 ERROR [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (org.ovirt.thread.pool-6-thread-47) [267b35df] Failed to migrate VM 'vpn1' 2017-10-01 14:55:12,246-04 INFO [org.ovirt.engine.core.bll.MigrateVmCommand] (org.ovirt.thread.pool-6-thread-47) [64591750] Lock Acquired to object 'EngineLock:{exclusiveLocks='[c8682859-9e55-460a-9ef0-d03e4a85f50e=VM]', sharedLocks=''}' 2017-10-01 14:55:12,367-04 INFO [org.ovirt.engine.core.bll.MigrateVmCommand] (org.ovirt.thread.pool-6-thread-47) [64591750] Running command: MigrateVmCommand internal: true. Entities affected : ID: c8682859-9e55-460a-9ef0-d03e4a85f50e Type: VMAction group MIGRATE_VM with role type USER 2017-10-01 14:55:12,454-04 INFO [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand] (org.ovirt.thread.pool-6-thread-47) [64591750] START, MigrateVDSCommand( MigrateVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vmId='c8682859-9e55-460a-9ef0-d03e4a85f50e', srcHost='virt1-mgmt', dstVdsId='a0ccfd11-3402-4eba-a119-4987561b738e', dstHost='virt2-mgmt:54321', migrationMethod='ONLINE', tunnelMigration='false', migrationDowntime='0', autoConverge='true', migrateCompressed='false', consoleAddress='null', maxBandwidth='null', enableGuestEvents='true', maxIncomingMigrations='2', maxOutgoingMigrations='2', convergenceSchedule='[init=[{name=setDowntime, params=[100]}], stalling=[{limit=1, action={name=setDowntime, params=[150]}}, {limit=2, action={name=setDowntime, params=[200]}}, {limit=3, action={name=setDowntime, params=[300]}}, {limit=4, action={name=setDowntime, params=[400]}}, {limit=6, action={name=setDowntime, params=[500]}}, {limit=-1, action={name=abort, params=[]}}]]'}), log id: 3bb73f13 2017-10-01 14:55:12,454-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] (org.ovirt.thread.pool-6-thread-47) [64591750] START, MigrateBrokerVDSCommand(HostName = virt1, MigrateVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vmId='c8682859-9e55-460a-9ef0-d03e4a85f50e', srcHost='virt1-mgmt', dstVdsId='a0ccfd11-3402-4eba-a119-4987561b738e', dstHost='virt2-mgmt:54321', migrationMethod='ONLINE', tunnelMigration='false', migrationDowntime='0', autoConverge='true', migrateCompressed='false', consoleAddress='null', maxBandwidth='null', enableGuestEvents='true', maxIncomingMigrations='2', maxOutgoingMigrations='2', convergenceSchedule='[init=[{name=setDowntime, params=[100]}], stalling=[{limit=1, action={name=setDowntime, params=[150]}}, {limit=2, action={name=setDowntime, params=[200]}}, {limit=3, action={name=setDowntime, params=[300]}}, {limit=4, action={name=setDowntime, params=[400]}}, {limit=6, action={name=setDowntime, params=[500]}}, {limit=-1, action={name=abort, params=[]}}]]'}), log id: 2f801854 2017-10-01 14:55:12,638-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [4ad0e7ed] START, ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 6bca8749 2017-10-01 14:55:12,638-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [4ad0e7ed] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 14:55:12,781-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] (org.ovirt.thread.pool-6-thread-47) [64591750] FINISH, MigrateBrokerVDSCommand, log id: 2f801854 2017-10-01 14:55:12,783-04 INFO [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand] (org.ovirt.thread.pool-6-thread-47) [64591750] FINISH, MigrateVDSCommand, return: MigratingFrom, log id: 3bb73f13 2017-10-01 14:55:12,788-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-47) [64591750] EVENT_ID: VM_MIGRATION_START_SYSTEM_INITIATED(67), Correlation ID: 64591750, Job ID: 8616c304-8cda-4d6e-ad83-b0ceb1e8c137, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Migration initiated by system (VM: dhcp1, Source: virt1, Destination: virt2, Reason: Host preparing for maintenance). 2017-10-01 14:55:12,804-04 INFO [org.ovirt.engine.core.bll.MigrateVmCommand] (org.ovirt.thread.pool-6-thread-47) [4db23c36] Lock Acquired to object 'EngineLock:{exclusiveLocks='[aa487207-7ff4-465a-9d9b-2a103d50dc77=VM]', sharedLocks=''}' 2017-10-01 14:55:12,879-04 INFO [org.ovirt.engine.core.bll.MigrateVmCommand] (org.ovirt.thread.pool-6-thread-47) [4db23c36] Running command: MigrateVmCommand internal: true. Entities affected : ID: aa487207-7ff4-465a-9d9b-2a103d50dc77 Type: VMAction group MIGRATE_VM with role type USER 2017-10-01 14:55:12,960-04 INFO [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand] (org.ovirt.thread.pool-6-thread-47) [4db23c36] START, MigrateVDSCommand( MigrateVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vmId='aa487207-7ff4-465a-9d9b-2a103d50dc77', srcHost='virt1-mgmt', dstVdsId='a0ccfd11-3402-4eba-a119-4987561b738e', dstHost='virt2-mgmt:54321', migrationMethod='ONLINE', tunnelMigration='false', migrationDowntime='0', autoConverge='true', migrateCompressed='false', consoleAddress='null', maxBandwidth='null', enableGuestEvents='true', maxIncomingMigrations='2', maxOutgoingMigrations='2', convergenceSchedule='[init=[{name=setDowntime, params=[100]}], stalling=[{limit=1, action={name=setDowntime, params=[150]}}, {limit=2, action={name=setDowntime, params=[200]}}, {limit=3, action={name=setDowntime, params=[300]}}, {limit=4, action={name=setDowntime, params=[400]}}, {limit=6, action={name=setDowntime, params=[500]}}, {limit=-1, action={name=abort, params=[]}}]]'}), log id: 1805860a 2017-10-01 14:55:12,960-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] (org.ovirt.thread.pool-6-thread-47) [4db23c36] START, MigrateBrokerVDSCommand(HostName = virt1, MigrateVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vmId='aa487207-7ff4-465a-9d9b-2a103d50dc77', srcHost='virt1-mgmt', dstVdsId='a0ccfd11-3402-4eba-a119-4987561b738e', dstHost='virt2-mgmt:54321', migrationMethod='ONLINE', tunnelMigration='false', migrationDowntime='0', autoConverge='true', migrateCompressed='false', consoleAddress='null', maxBandwidth='null', enableGuestEvents='true', maxIncomingMigrations='2', maxOutgoingMigrations='2', convergenceSchedule='[init=[{name=setDowntime, params=[100]}], stalling=[{limit=1, action={name=setDowntime, params=[150]}}, {limit=2, action={name=setDowntime, params=[200]}}, {limit=3, action={name=setDowntime, params=[300]}}, {limit=4, action={name=setDowntime, params=[400]}}, {limit=6, action={name=setDowntime, params=[500]}}, {limit=-1, action={name=abort, params=[]}}]]'}), log id: 3dec8680 2017-10-01 14:55:12,961-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [4ad0e7ed] FINISH, ConnectStoragePoolVDSCommand, log id: 6bca8749 2017-10-01 14:55:13,970-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] (org.ovirt.thread.pool-6-thread-47) [4db23c36] FINISH, MigrateBrokerVDSCommand, log id: 3dec8680 2017-10-01 14:55:13,970-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler2) [4ad0e7ed] hostFromVds::selectedVds - 'virt4', spmStatus 'Free', storage pool 'EECS-Primary', storage pool version '4.1' 2017-10-01 14:55:13,971-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler2) [4ad0e7ed] Found spm host 'a0ccfd11-3402-4eba-a119-4987561b738e', host name: 'virt2', according to spmId: '3'. 2017-10-01 14:55:13,972-04 INFO [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand] (org.ovirt.thread.pool-6-thread-47) [4db23c36] FINISH, MigrateVDSCommand, return: MigratingFrom, log id: 1805860a 2017-10-01 14:55:13,977-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-47) [4db23c36] EVENT_ID: VM_MIGRATION_START_SYSTEM_INITIATED(67), Correlation ID: 4db23c36, Job ID: fada4f20-3260-460c-879a-f1b7807047c8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Migration initiated by system (VM: ftp1, Source: virt1, Destination: virt2, Reason: Host preparing for maintenance). 2017-10-01 14:55:13,988-04 INFO [org.ovirt.engine.core.bll.MigrateVmCommand] (org.ovirt.thread.pool-6-thread-47) [2346e2d3] Lock Acquired to object 'EngineLock:{exclusiveLocks='[bfe9a524-2238-4bab-a91f-5dc6ed270896=VM]', sharedLocks=''}' 2017-10-01 14:55:14,063-04 INFO [org.ovirt.engine.core.bll.MigrateVmCommand] (org.ovirt.thread.pool-6-thread-47) [2346e2d3] Running command: MigrateVmCommand internal: true. Entities affected : ID: bfe9a524-2238-4bab-a91f-5dc6ed270896 Type: VMAction group MIGRATE_VM with role type USER 2017-10-01 14:55:14,140-04 INFO [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand] (org.ovirt.thread.pool-6-thread-47) [2346e2d3] START, MigrateVDSCommand( MigrateVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vmId='bfe9a524-2238-4bab-a91f-5dc6ed270896', srcHost='virt1-mgmt', dstVdsId='a0ccfd11-3402-4eba-a119-4987561b738e', dstHost='virt2-mgmt:54321', migrationMethod='ONLINE', tunnelMigration='false', migrationDowntime='0', autoConverge='true', migrateCompressed='false', consoleAddress='null', maxBandwidth='null', enableGuestEvents='true', maxIncomingMigrations='2', maxOutgoingMigrations='2', convergenceSchedule='[init=[{name=setDowntime, params=[100]}], stalling=[{limit=1, action={name=setDowntime, params=[150]}}, {limit=2, action={name=setDowntime, params=[200]}}, {limit=3, action={name=setDowntime, params=[300]}}, {limit=4, action={name=setDowntime, params=[400]}}, {limit=6, action={name=setDowntime, params=[500]}}, {limit=-1, action={name=abort, params=[]}}]]'}), log id: 563ea2e9 2017-10-01 14:55:14,140-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] (org.ovirt.thread.pool-6-thread-47) [2346e2d3] START, MigrateBrokerVDSCommand(HostName = virt1, MigrateVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vmId='bfe9a524-2238-4bab-a91f-5dc6ed270896', srcHost='virt1-mgmt', dstVdsId='a0ccfd11-3402-4eba-a119-4987561b738e', dstHost='virt2-mgmt:54321', migrationMethod='ONLINE', tunnelMigration='false', migrationDowntime='0', autoConverge='true', migrateCompressed='false', consoleAddress='null', maxBandwidth='null', enableGuestEvents='true', maxIncomingMigrations='2', maxOutgoingMigrations='2', convergenceSchedule='[init=[{name=setDowntime, params=[100]}], stalling=[{limit=1, action={name=setDowntime, params=[150]}}, {limit=2, action={name=setDowntime, params=[200]}}, {limit=3, action={name=setDowntime, params=[300]}}, {limit=4, action={name=setDowntime, params=[400]}}, {limit=6, action={name=setDowntime, params=[500]}}, {limit=-1, action={name=abort, params=[]}}]]'}), log id: 1b9eaaf1 2017-10-01 14:55:14,141-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler2) [4ad0e7ed] SpmStatus on vds 'a0ccfd11-3402-4eba-a119-4987561b738e': 'Free' 2017-10-01 14:55:14,143-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler2) [4ad0e7ed] starting spm on vds 'virt4', storage pool 'EECS-Primary', prevId '3', LVER '58' 2017-10-01 14:55:14,144-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler2) [4ad0e7ed] START, SpmStartVDSCommand(HostName = virt4, SpmStartVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', prevId='3', prevLVER='58', storagePoolFormatType='V4', recoveryMode='Manual', SCSIFencing='false'}), log id: 7605afbb 2017-10-01 14:55:14,885-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] (org.ovirt.thread.pool-6-thread-47) [2346e2d3] FINISH, MigrateBrokerVDSCommand, log id: 1b9eaaf1 2017-10-01 14:55:14,885-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler2) [4ad0e7ed] spmStart polling started: taskId '769f493a-8059-44f3-b2ad-c2349342771d' 2017-10-01 14:55:14,886-04 INFO [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand] (org.ovirt.thread.pool-6-thread-47) [2346e2d3] FINISH, MigrateVDSCommand, return: MigratingFrom, log id: 563ea2e9 2017-10-01 14:55:14,891-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-47) [2346e2d3] EVENT_ID: VM_MIGRATION_START_SYSTEM_INITIATED(67), Correlation ID: 2346e2d3, Job ID: e47484f4-ac65-4022-8f7c-4f9a883bdc9d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Migration initiated by system (VM: pcmode, Source: virt1, Destination: virt2, Reason: Host preparing for maintenance). 2017-10-01 14:55:14,912-04 INFO [org.ovirt.engine.core.bll.MigrateVmCommand] (org.ovirt.thread.pool-6-thread-47) [325def75] Lock Acquired to object 'EngineLock:{exclusiveLocks='[1325cebf-834e-4571-a3e1-82beb0021ffe=VM]', sharedLocks=''}' 2017-10-01 14:55:14,988-04 INFO [org.ovirt.engine.core.bll.MigrateVmCommand] (org.ovirt.thread.pool-6-thread-47) [325def75] Running command: MigrateVmCommand internal: true. Entities affected : ID: 1325cebf-834e-4571-a3e1-82beb0021ffe Type: VMAction group MIGRATE_VM with role type USER 2017-10-01 14:55:15,067-04 INFO [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand] (org.ovirt.thread.pool-6-thread-47) [325def75] START, MigrateVDSCommand( MigrateVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vmId='1325cebf-834e-4571-a3e1-82beb0021ffe', srcHost='virt1-mgmt', dstVdsId='a0ccfd11-3402-4eba-a119-4987561b738e', dstHost='virt2-mgmt:54321', migrationMethod='ONLINE', tunnelMigration='false', migrationDowntime='0', autoConverge='true', migrateCompressed='false', consoleAddress='null', maxBandwidth='null', enableGuestEvents='true', maxIncomingMigrations='2', maxOutgoingMigrations='2', convergenceSchedule='[init=[{name=setDowntime, params=[100]}], stalling=[{limit=1, action={name=setDowntime, params=[150]}}, {limit=2, action={name=setDowntime, params=[200]}}, {limit=3, action={name=setDowntime, params=[300]}}, {limit=4, action={name=setDowntime, params=[400]}}, {limit=6, action={name=setDowntime, params=[500]}}, {limit=-1, action={name=abort, params=[]}}]]'}), log id: 3a93d9db 2017-10-01 14:55:15,067-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] (org.ovirt.thread.pool-6-thread-47) [325def75] START, MigrateBrokerVDSCommand(HostName = virt1, MigrateVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vmId='1325cebf-834e-4571-a3e1-82beb0021ffe', srcHost='virt1-mgmt', dstVdsId='a0ccfd11-3402-4eba-a119-4987561b738e', dstHost='virt2-mgmt:54321', migrationMethod='ONLINE', tunnelMigration='false', migrationDowntime='0', autoConverge='true', migrateCompressed='false', consoleAddress='null', maxBandwidth='null', enableGuestEvents='true', maxIncomingMigrations='2', maxOutgoingMigrations='2', convergenceSchedule='[init=[{name=setDowntime, params=[100]}], stalling=[{limit=1, action={name=setDowntime, params=[150]}}, {limit=2, action={name=setDowntime, params=[200]}}, {limit=3, action={name=setDowntime, params=[300]}}, {limit=4, action={name=setDowntime, params=[400]}}, {limit=6, action={name=setDowntime, params=[500]}}, {limit=-1, action={name=abort, params=[]}}]]'}), log id: 34870fca 2017-10-01 14:55:15,889-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] (org.ovirt.thread.pool-6-thread-47) [325def75] FINISH, MigrateBrokerVDSCommand, log id: 34870fca 2017-10-01 14:55:15,891-04 INFO [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand] (org.ovirt.thread.pool-6-thread-47) [325def75] FINISH, MigrateVDSCommand, return: MigratingFrom, log id: 3a93d9db 2017-10-01 14:55:15,896-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-47) [325def75] EVENT_ID: VM_MIGRATION_START_SYSTEM_INITIATED(67), Correlation ID: 325def75, Job ID: f694cca6-07c0-4595-82f7-155e1765e0eb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Migration initiated by system (VM: dns1, Source: virt1, Destination: virt2, Reason: Host preparing for maintenance). 2017-10-01 14:55:15,913-04 INFO [org.ovirt.engine.core.bll.MigrateVmCommand] (org.ovirt.thread.pool-6-thread-47) [34bc2b81] Lock Acquired to object 'EngineLock:{exclusiveLocks='[db3e08fb-c111-442d-aeec-1507da30f486=VM]', sharedLocks=''}' 2017-10-01 14:55:15,988-04 INFO [org.ovirt.engine.core.bll.MigrateVmCommand] (org.ovirt.thread.pool-6-thread-47) [34bc2b81] Running command: MigrateVmCommand internal: true. Entities affected : ID: db3e08fb-c111-442d-aeec-1507da30f486 Type: VMAction group MIGRATE_VM with role type USER 2017-10-01 14:55:16,067-04 INFO [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand] (org.ovirt.thread.pool-6-thread-47) [34bc2b81] START, MigrateVDSCommand( MigrateVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vmId='db3e08fb-c111-442d-aeec-1507da30f486', srcHost='virt1-mgmt', dstVdsId='a0ccfd11-3402-4eba-a119-4987561b738e', dstHost='virt2-mgmt:54321', migrationMethod='ONLINE', tunnelMigration='false', migrationDowntime='0', autoConverge='true', migrateCompressed='false', consoleAddress='null', maxBandwidth='null', enableGuestEvents='true', maxIncomingMigrations='2', maxOutgoingMigrations='2', convergenceSchedule='[init=[{name=setDowntime, params=[100]}], stalling=[{limit=1, action={name=setDowntime, params=[150]}}, {limit=2, action={name=setDowntime, params=[200]}}, {limit=3, action={name=setDowntime, params=[300]}}, {limit=4, action={name=setDowntime, params=[400]}}, {limit=6, action={name=setDowntime, params=[500]}}, {limit=-1, action={name=abort, params=[]}}]]'}), log id: 2f966eee 2017-10-01 14:55:16,068-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] (org.ovirt.thread.pool-6-thread-47) [34bc2b81] START, MigrateBrokerVDSCommand(HostName = virt1, MigrateVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vmId='db3e08fb-c111-442d-aeec-1507da30f486', srcHost='virt1-mgmt', dstVdsId='a0ccfd11-3402-4eba-a119-4987561b738e', dstHost='virt2-mgmt:54321', migrationMethod='ONLINE', tunnelMigration='false', migrationDowntime='0', autoConverge='true', migrateCompressed='false', consoleAddress='null', maxBandwidth='null', enableGuestEvents='true', maxIncomingMigrations='2', maxOutgoingMigrations='2', convergenceSchedule='[init=[{name=setDowntime, params=[100]}], stalling=[{limit=1, action={name=setDowntime, params=[150]}}, {limit=2, action={name=setDowntime, params=[200]}}, {limit=3, action={name=setDowntime, params=[300]}}, {limit=4, action={name=setDowntime, params=[400]}}, {limit=6, action={name=setDowntime, params=[500]}}, {limit=-1, action={name=abort, params=[]}}]]'}), log id: 672aaa2b 2017-10-01 14:55:17,024-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] (org.ovirt.thread.pool-6-thread-47) [34bc2b81] FINISH, MigrateBrokerVDSCommand, log id: 672aaa2b 2017-10-01 14:55:17,026-04 INFO [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand] (org.ovirt.thread.pool-6-thread-47) [34bc2b81] FINISH, MigrateVDSCommand, return: MigratingFrom, log id: 2f966eee 2017-10-01 14:55:17,033-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-47) [34bc2b81] EVENT_ID: VM_MIGRATION_START_SYSTEM_INITIATED(67), Correlation ID: 34bc2b81, Job ID: 869f0fd4-4e10-4241-b2c0-d948f9643fbf, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Migration initiated by system (VM: print, Source: virt1, Destination: virt2, Reason: Host preparing for maintenance). 2017-10-01 14:55:17,050-04 INFO [org.ovirt.engine.core.bll.MigrateVmCommand] (org.ovirt.thread.pool-6-thread-47) [4232b55d] Lock Acquired to object 'EngineLock:{exclusiveLocks='[ddf37076-e795-4c87-a4fb-9efcaae76d72=VM]', sharedLocks=''}' 2017-10-01 14:55:17,126-04 INFO [org.ovirt.engine.core.bll.MigrateVmCommand] (org.ovirt.thread.pool-6-thread-47) [4232b55d] Running command: MigrateVmCommand internal: true. Entities affected : ID: ddf37076-e795-4c87-a4fb-9efcaae76d72 Type: VMAction group MIGRATE_VM with role type USER 2017-10-01 14:55:17,206-04 INFO [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand] (org.ovirt.thread.pool-6-thread-47) [4232b55d] START, MigrateVDSCommand( MigrateVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vmId='ddf37076-e795-4c87-a4fb-9efcaae76d72', srcHost='virt1-mgmt', dstVdsId='a0ccfd11-3402-4eba-a119-4987561b738e', dstHost='virt2-mgmt:54321', migrationMethod='ONLINE', tunnelMigration='false', migrationDowntime='0', autoConverge='true', migrateCompressed='false', consoleAddress='null', maxBandwidth='null', enableGuestEvents='true', maxIncomingMigrations='2', maxOutgoingMigrations='2', convergenceSchedule='[init=[{name=setDowntime, params=[100]}], stalling=[{limit=1, action={name=setDowntime, params=[150]}}, {limit=2, action={name=setDowntime, params=[200]}}, {limit=3, action={name=setDowntime, params=[300]}}, {limit=4, action={name=setDowntime, params=[400]}}, {limit=6, action={name=setDowntime, params=[500]}}, {limit=-1, action={name=abort, params=[]}}]]'}), log id: 725a6d16 2017-10-01 14:55:17,206-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] (org.ovirt.thread.pool-6-thread-47) [4232b55d] START, MigrateBrokerVDSCommand(HostName = virt1, MigrateVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vmId='ddf37076-e795-4c87-a4fb-9efcaae76d72', srcHost='virt1-mgmt', dstVdsId='a0ccfd11-3402-4eba-a119-4987561b738e', dstHost='virt2-mgmt:54321', migrationMethod='ONLINE', tunnelMigration='false', migrationDowntime='0', autoConverge='true', migrateCompressed='false', consoleAddress='null', maxBandwidth='null', enableGuestEvents='true', maxIncomingMigrations='2', maxOutgoingMigrations='2', convergenceSchedule='[init=[{name=setDowntime, params=[100]}], stalling=[{limit=1, action={name=setDowntime, params=[150]}}, {limit=2, action={name=setDowntime, params=[200]}}, {limit=3, action={name=setDowntime, params=[300]}}, {limit=4, action={name=setDowntime, params=[400]}}, {limit=6, action={name=setDowntime, params=[500]}}, {limit=-1, action={name=abort, params=[]}}]]'}), log id: 650be7a6 2017-10-01 14:55:18,211-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] (org.ovirt.thread.pool-6-thread-47) [4232b55d] FINISH, MigrateBrokerVDSCommand, log id: 650be7a6 2017-10-01 14:55:18,213-04 INFO [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand] (org.ovirt.thread.pool-6-thread-47) [4232b55d] FINISH, MigrateVDSCommand, return: MigratingFrom, log id: 725a6d16 2017-10-01 14:55:18,225-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-47) [4232b55d] EVENT_ID: VM_MIGRATION_START_SYSTEM_INITIATED(67), Correlation ID: 4232b55d, Job ID: 1e73de3d-36b7-4b23-9a56-735f7ff910d6, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Migration initiated by system (VM: dc1, Source: virt1, Destination: virt2, Reason: Host preparing for maintenance). 2017-10-01 14:55:22,073-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmsStatisticsFetcher] (DefaultQuartzScheduler6) [1cbde662] Fetched 6 VMs from VDS 'a0ccfd11-3402-4eba-a119-4987561b738e' 2017-10-01 14:55:22,074-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler6) [1cbde662] VM 'c8682859-9e55-460a-9ef0-d03e4a85f50e'(dhcp1) was unexpectedly detected as 'MigratingTo' on VDS 'a0ccfd11-3402-4eba-a119-4987561b738e'(virt2) (expected on '45f8b331-842e-48e7-9df8-56adddb93836') 2017-10-01 14:55:22,074-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler6) [1cbde662] VM 'c8682859-9e55-460a-9ef0-d03e4a85f50e' is migrating to VDS 'a0ccfd11-3402-4eba-a119-4987561b738e'(virt2) ignoring it in the refresh until migration is done 2017-10-01 14:55:22,092-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmsStatisticsFetcher] (DefaultQuartzScheduler9) [901457c] Fetched 11 VMs from VDS '5ae29ed2-9c0f-4754-8df3-b43d14726846' 2017-10-01 14:55:22,092-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler9) [901457c] VM '911d6883-607c-464e-b46b-8e6141bdd2de'(hdb) was unexpectedly detected as 'MigratingTo' on VDS '5ae29ed2-9c0f-4754-8df3-b43d14726846'(virt4) (expected on '45f8b331-842e-48e7-9df8-56adddb93836') 2017-10-01 14:55:22,092-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler9) [901457c] VM '911d6883-607c-464e-b46b-8e6141bdd2de' is migrating to VDS '5ae29ed2-9c0f-4754-8df3-b43d14726846'(virt4) ignoring it in the refresh until migration is done 2017-10-01 14:55:22,093-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler9) [901457c] VM '1c7e966f-29e8-4788-b64c-86e52f415061'(udb) was unexpectedly detected as 'MigratingTo' on VDS '5ae29ed2-9c0f-4754-8df3-b43d14726846'(virt4) (expected on 'e1e71e9d-a0cd-47db-96c6-f724f6cea736') 2017-10-01 14:55:22,093-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler9) [901457c] VM '1c7e966f-29e8-4788-b64c-86e52f415061' is migrating to VDS '5ae29ed2-9c0f-4754-8df3-b43d14726846'(virt4) ignoring it in the refresh until migration is done 2017-10-01 14:55:22,224-04 WARN [org.ovirt.engine.core.vdsbroker.monitoring.VmsMonitoring] (DefaultQuartzScheduler5) [6c79d71b] skipping VM '1c7e966f-29e8-4788-b64c-86e52f415061' from this monitoring cycle - the VM data has changed since fetching the data 2017-10-01 14:55:24,434-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler2) [4ad0e7ed] spmStart polling ended: taskId '769f493a-8059-44f3-b2ad-c2349342771d' task status 'finished' 2017-10-01 14:55:24,442-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler2) [4ad0e7ed] spmStart polling ended, spm status: SPM 2017-10-01 14:55:24,442-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler2) [4ad0e7ed] START, HSMClearTaskVDSCommand(HostName = virt4, HSMTaskGuidBaseVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', taskId='769f493a-8059-44f3-b2ad-c2349342771d'}), log id: 2638dd92 2017-10-01 14:55:24,512-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler2) [4ad0e7ed] FINISH, HSMClearTaskVDSCommand, log id: 2638dd92 2017-10-01 14:55:24,512-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler2) [4ad0e7ed] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6bd1ad21, log id: 7605afbb 2017-10-01 14:55:24,513-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler2) [4ad0e7ed] Initialize Irs proxy from vds: virt4-mgmt 2017-10-01 14:55:24,533-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [4ad0e7ed] EVENT_ID: IRS_HOSTED_ON_VDS(204), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host virt4 (Address: virt4-mgmt). 2017-10-01 14:55:24,534-04 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to virt4-mgmt/192.168.0.37 2017-10-01 14:55:24,542-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-6-thread-17) [4ad0e7ed] START, SPMGetAllTasksInfoVDSCommand( IrsBaseVDSCommandParameters:{runAsync='true', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', ignoreFailoverLimit='false'}), log id: 7cd4fb50 2017-10-01 14:55:25,281-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-6-thread-17) [4ad0e7ed] -- executeIrsBrokerCommand: Attempting on storage pool 'aca61eac-ed8a-484b-bbee-3645148f6405' 2017-10-01 14:55:25,281-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-6-thread-17) [4ad0e7ed] START, HSMGetAllTasksInfoVDSCommand(HostName = virt4, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846'}), log id: 1d98dc52 2017-10-01 14:55:26,285-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-6-thread-17) [4ad0e7ed] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 1d98dc52 2017-10-01 14:55:26,285-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-6-thread-17) [4ad0e7ed] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 7cd4fb50 2017-10-01 14:55:26,285-04 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-6-thread-17) [4ad0e7ed] Discovered no tasks on Storage Pool 'EECS-Primary' 2017-10-01 14:55:30,092-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-11) [] VM 'c8682859-9e55-460a-9ef0-d03e4a85f50e'(dhcp1) was unexpectedly detected as 'Up' on VDS 'a0ccfd11-3402-4eba-a119-4987561b738e'(virt2) (expected on '45f8b331-842e-48e7-9df8-56adddb93836') 2017-10-01 14:55:30,099-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ForkJoinPool-1-worker-11) [] EVENT_ID: VM_MIGRATION_DONE(63), Correlation ID: 64591750, Job ID: 8616c304-8cda-4d6e-ad83-b0ceb1e8c137, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Migration completed (VM: dhcp1, Source: virt1, Destination: virt2, Duration: 17 seconds, Total: 17 seconds, Actual downtime: 53ms) 2017-10-01 14:55:30,102-04 INFO [org.ovirt.engine.core.bll.MigrateVmCommand] (ForkJoinPool-1-worker-11) [] Lock freed to object 'EngineLock:{exclusiveLocks='[c8682859-9e55-460a-9ef0-d03e4a85f50e=VM]', sharedLocks=''}' 2017-10-01 14:55:30,103-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVDSCommand] (ForkJoinPool-1-worker-11) [] START, FullListVDSCommand(HostName = virt2, FullListVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vmIds='[c8682859-9e55-460a-9ef0-d03e4a85f50e]'}), log id: 48d2e9a2 2017-10-01 14:55:30,154-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVDSCommand] (ForkJoinPool-1-worker-11) [] FINISH, FullListVDSCommand, return: [{acpiEnable=true, emulatedMachine=pc-i440fx-rhel7.3.0, afterMigrationStatus=, pid=36692, memGuaranteedSize=682, transparentHugePages=true, timeOffset=-14399, cpuType=SandyBridge, smp=1, guestNumaNodes=[Ljava.lang.Object;@6df889fd, smartcardEnable=false, custom={device_3956097a-819f-44bb-88fe-d2c5a0906e41device_5b17ce83-837a-4d64-892d-808d9e806531device_7bcc59ca-6ad6-4d23-a4ee-9322ad491ab5device_22e12242-2f31-407b-9aa0-bb8ccd743060=VmDevice:{id='VmDeviceId:{deviceId='22e12242-2f31-407b-9aa0-bb8ccd743060', vmId='c8682859-9e55-460a-9ef0-d03e4a85f50e'}', device='spicevmc', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=3}', managed='false', plugged='true', readOnly='false', deviceAlias='channel2', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_3956097a-819f-44bb-88fe-d2c5a0906e41device_5b17ce83-837a-4d64-892d-808d9e806531=VmDevice:{id='VmDeviceId:{deviceId='5b17ce83-837a-4d64-892d-808d9e806531', vmId='c8682859-9e55-460a-9ef0-d03e4a85f50e'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=1}', managed='false', plugged='true', readOnly='false', deviceAlias='channel0', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_3956097a-819f-44bb-88fe-d2c5a0906e41=VmDevice:{id='VmDeviceId:{deviceId='3956097a-819f-44bb-88fe-d2c5a0906e41', vmId='c8682859-9e55-460a-9ef0-d03e4a85f50e'}', device='ide', type='CONTROLLER', bootOrder='0', specParams='[]', address='{slot=0x01, bus=0x00, domain=0x0000, type=pci, function=0x1}', managed='false', plugged='true', readOnly='false', deviceAlias='ide', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_3956097a-819f-44bb-88fe-d2c5a0906e41device_5b17ce83-837a-4d64-892d-808d9e806531device_7bcc59ca-6ad6-4d23-a4ee-9322ad491ab5=VmDevice:{id='VmDeviceId:{deviceId='7bcc59ca-6ad6-4d23-a4ee-9322ad491ab5', vmId='c8682859-9e55-460a-9ef0-d03e4a85f50e'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=2}', managed='false', plugged='true', readOnly='false', deviceAlias='channel1', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}}, vmType=kvm, memSize=1024, smpCoresPerSocket=1, vmName=dhcp1, nice=0, status=Up, maxMemSize=4096, bootMenuEnable=false, vmId=c8682859-9e55-460a-9ef0-d03e4a85f50e, smpThreadsPerCore=1, guestDiskMapping={}, kvmEnable=true, displayNetwork=yorku, devices=[Ljava.lang.Object;@58ae058d, display=qxl, maxVCpus=16, clientIp=, statusTime=4720813340, maxMemSlots=16}], log id: 48d2e9a2 2017-10-01 14:55:30,158-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (ForkJoinPool-1-worker-11) [] Received a memballoon Device without an address when processing VM c8682859-9e55-460a-9ef0-d03e4a85f50e devices, skipping device: {device=memballoon, specParams={model=none}, type=balloon, target=1048576, alias=balloon0} 2017-10-01 14:55:30,158-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (ForkJoinPool-1-worker-11) [] Received a spice Device without an address when processing VM c8682859-9e55-460a-9ef0-d03e4a85f50e devices, skipping device: {device=spice, specParams={fileTransferEnable=true, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,ssmartcard,susbredir, copyPasteEnable=true, displayIp=130.63.94.35, displayNetwork=yorku}, type=graphics, deviceId=71bf73f5-1c73-4e2a-8dea-0cb9402d5e4e, tlsPort=5903} 2017-10-01 14:55:37,097-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmsStatisticsFetcher] (DefaultQuartzScheduler5) [9e3020f] Fetched 7 VMs from VDS 'a0ccfd11-3402-4eba-a119-4987561b738e' 2017-10-01 14:55:37,097-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler5) [9e3020f] VM 'ae56c31c-71f7-433a-b198-a1c2c846325e'(forum) moved from 'NotResponding' --> 'Up' 2017-10-01 14:55:37,098-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler5) [9e3020f] VM 'aa487207-7ff4-465a-9d9b-2a103d50dc77'(ftp1) was unexpectedly detected as 'MigratingTo' on VDS 'a0ccfd11-3402-4eba-a119-4987561b738e'(virt2) (expected on '45f8b331-842e-48e7-9df8-56adddb93836') 2017-10-01 14:55:37,098-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler5) [9e3020f] VM 'aa487207-7ff4-465a-9d9b-2a103d50dc77' is migrating to VDS 'a0ccfd11-3402-4eba-a119-4987561b738e'(virt2) ignoring it in the refresh until migration is done 2017-10-01 14:55:37,098-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler5) [9e3020f] VM '4ba80442-b361-4026-ae8c-895fd28042b0'(dist) moved from 'NotResponding' --> 'Up' 2017-10-01 14:55:37,228-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmsStatisticsFetcher] (DefaultQuartzScheduler7) [5c020e6c] Fetched 12 VMs from VDS '5ae29ed2-9c0f-4754-8df3-b43d14726846' 2017-10-01 14:55:37,229-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler7) [5c020e6c] VM '911d6883-607c-464e-b46b-8e6141bdd2de'(hdb) was unexpectedly detected as 'Up' on VDS '5ae29ed2-9c0f-4754-8df3-b43d14726846'(virt4) (expected on '45f8b331-842e-48e7-9df8-56adddb93836') 2017-10-01 14:55:37,229-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler7) [5c020e6c] VM '8391f9a7-7fe7-4025-8fba-6273ae587a52'(archive) was unexpectedly detected as 'MigratingTo' on VDS '5ae29ed2-9c0f-4754-8df3-b43d14726846'(virt4) (expected on '45f8b331-842e-48e7-9df8-56adddb93836') 2017-10-01 14:55:37,229-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler7) [5c020e6c] VM '8391f9a7-7fe7-4025-8fba-6273ae587a52' is migrating to VDS '5ae29ed2-9c0f-4754-8df3-b43d14726846'(virt4) ignoring it in the refresh until migration is done 2017-10-01 14:55:37,229-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler7) [5c020e6c] VM '1c7e966f-29e8-4788-b64c-86e52f415061'(udb) was unexpectedly detected as 'MigratingTo' on VDS '5ae29ed2-9c0f-4754-8df3-b43d14726846'(virt4) (expected on 'e1e71e9d-a0cd-47db-96c6-f724f6cea736') 2017-10-01 14:55:37,229-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler7) [5c020e6c] VM '1c7e966f-29e8-4788-b64c-86e52f415061' is migrating to VDS '5ae29ed2-9c0f-4754-8df3-b43d14726846'(virt4) ignoring it in the refresh until migration is done 2017-10-01 14:55:37,238-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [5c020e6c] EVENT_ID: VM_MIGRATION_DONE(63), Correlation ID: 4e6b2aa0, Job ID: b2979192-145f-4a51-a56d-7bdefc27d56a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Migration completed (VM: hdb, Source: virt1, Destination: virt4, Duration: 27 seconds, Total: 27 seconds, Actual downtime: 49ms) 2017-10-01 14:55:37,241-04 INFO [org.ovirt.engine.core.bll.MigrateVmCommand] (DefaultQuartzScheduler7) [5c020e6c] Lock freed to object 'EngineLock:{exclusiveLocks='[911d6883-607c-464e-b46b-8e6141bdd2de=VM]', sharedLocks=''}' 2017-10-01 14:55:38,242-04 WARN [org.ovirt.engine.core.vdsbroker.monitoring.VmsMonitoring] (DefaultQuartzScheduler8) [3fedabe2] skipping VM '1c7e966f-29e8-4788-b64c-86e52f415061' from this monitoring cycle - the VM data has changed since fetching the data 2017-10-01 14:55:39,651-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler1) [51338b2a] VM '911d6883-607c-464e-b46b-8e6141bdd2de' was reported as Down on VDS '45f8b331-842e-48e7-9df8-56adddb93836'(virt1) 2017-10-01 14:55:39,651-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (DefaultQuartzScheduler1) [51338b2a] START, DestroyVDSCommand(HostName = virt1, DestroyVmVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vmId='911d6883-607c-464e-b46b-8e6141bdd2de', force='false', secondsToWait='0', gracefully='false', reason='', ignoreNoVm='true'}), log id: 762620ad 2017-10-01 14:55:40,095-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (DefaultQuartzScheduler1) [51338b2a] FINISH, DestroyVDSCommand, log id: 762620ad 2017-10-01 14:55:40,095-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler1) [51338b2a] VM '911d6883-607c-464e-b46b-8e6141bdd2de'(hdb) was unexpectedly detected as 'Down' on VDS '45f8b331-842e-48e7-9df8-56adddb93836'(virt1) (expected on '5ae29ed2-9c0f-4754-8df3-b43d14726846') 2017-10-01 14:55:40,096-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler1) [51338b2a] VM 'c8682859-9e55-460a-9ef0-d03e4a85f50e' was reported as Down on VDS '45f8b331-842e-48e7-9df8-56adddb93836'(virt1) 2017-10-01 14:55:40,096-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (DefaultQuartzScheduler1) [51338b2a] START, DestroyVDSCommand(HostName = virt1, DestroyVmVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vmId='c8682859-9e55-460a-9ef0-d03e4a85f50e', force='false', secondsToWait='0', gracefully='false', reason='', ignoreNoVm='true'}), log id: f219a55 2017-10-01 14:55:40,394-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (DefaultQuartzScheduler1) [51338b2a] FINISH, DestroyVDSCommand, log id: f219a55 2017-10-01 14:55:40,394-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler1) [51338b2a] VM 'c8682859-9e55-460a-9ef0-d03e4a85f50e'(dhcp1) was unexpectedly detected as 'Down' on VDS '45f8b331-842e-48e7-9df8-56adddb93836'(virt1) (expected on 'a0ccfd11-3402-4eba-a119-4987561b738e') 2017-10-01 14:55:52,243-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmsStatisticsFetcher] (DefaultQuartzScheduler7) [411984f5] Fetched 8 VMs from VDS 'a0ccfd11-3402-4eba-a119-4987561b738e' 2017-10-01 14:55:52,244-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler7) [411984f5] VM 'bfe9a524-2238-4bab-a91f-5dc6ed270896'(pcmode) was unexpectedly detected as 'MigratingTo' on VDS 'a0ccfd11-3402-4eba-a119-4987561b738e'(virt2) (expected on '45f8b331-842e-48e7-9df8-56adddb93836') 2017-10-01 14:55:52,244-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler7) [411984f5] VM 'bfe9a524-2238-4bab-a91f-5dc6ed270896' is migrating to VDS 'a0ccfd11-3402-4eba-a119-4987561b738e'(virt2) ignoring it in the refresh until migration is done 2017-10-01 14:55:52,246-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler7) [411984f5] VM 'aa487207-7ff4-465a-9d9b-2a103d50dc77'(ftp1) was unexpectedly detected as 'MigratingTo' on VDS 'a0ccfd11-3402-4eba-a119-4987561b738e'(virt2) (expected on '45f8b331-842e-48e7-9df8-56adddb93836') 2017-10-01 14:55:52,246-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler7) [411984f5] VM 'aa487207-7ff4-465a-9d9b-2a103d50dc77' is migrating to VDS 'a0ccfd11-3402-4eba-a119-4987561b738e'(virt2) ignoring it in the refresh until migration is done 2017-10-01 14:55:52,251-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmsStatisticsFetcher] (DefaultQuartzScheduler4) [4e42cf4a] Fetched 13 VMs from VDS '5ae29ed2-9c0f-4754-8df3-b43d14726846' 2017-10-01 14:55:52,251-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler4) [4e42cf4a] VM '8391f9a7-7fe7-4025-8fba-6273ae587a52'(archive) was unexpectedly detected as 'Up' on VDS '5ae29ed2-9c0f-4754-8df3-b43d14726846'(virt4) (expected on '45f8b331-842e-48e7-9df8-56adddb93836') 2017-10-01 14:55:52,252-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler4) [4e42cf4a] VM '1c7e966f-29e8-4788-b64c-86e52f415061'(udb) was unexpectedly detected as 'MigratingTo' on VDS '5ae29ed2-9c0f-4754-8df3-b43d14726846'(virt4) (expected on 'e1e71e9d-a0cd-47db-96c6-f724f6cea736') 2017-10-01 14:55:52,252-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler4) [4e42cf4a] VM '1c7e966f-29e8-4788-b64c-86e52f415061' is migrating to VDS '5ae29ed2-9c0f-4754-8df3-b43d14726846'(virt4) ignoring it in the refresh until migration is done 2017-10-01 14:55:52,252-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler4) [4e42cf4a] VM '9ea2821e-557f-45cb-9872-1e6c5aeaf246'(rs) was unexpectedly detected as 'MigratingTo' on VDS '5ae29ed2-9c0f-4754-8df3-b43d14726846'(virt4) (expected on 'e1e71e9d-a0cd-47db-96c6-f724f6cea736') 2017-10-01 14:55:52,252-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler4) [4e42cf4a] VM '9ea2821e-557f-45cb-9872-1e6c5aeaf246' is migrating to VDS '5ae29ed2-9c0f-4754-8df3-b43d14726846'(virt4) ignoring it in the refresh until migration is done 2017-10-01 14:55:52,260-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [4e42cf4a] EVENT_ID: VM_MIGRATION_DONE(63), Correlation ID: 2d950fdc, Job ID: 332db2e9-620c-4c11-a225-018a7ad89be1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Migration completed (VM: archive, Source: virt1, Destination: virt4, Duration: 42 seconds, Total: 42 seconds, Actual downtime: 42ms) 2017-10-01 14:55:52,263-04 INFO [org.ovirt.engine.core.bll.MigrateVmCommand] (DefaultQuartzScheduler4) [4e42cf4a] Lock freed to object 'EngineLock:{exclusiveLocks='[8391f9a7-7fe7-4025-8fba-6273ae587a52=VM]', sharedLocks=''}' 2017-10-01 14:55:52,264-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVDSCommand] (DefaultQuartzScheduler4) [4e42cf4a] START, FullListVDSCommand(HostName = virt4, FullListVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vmIds='[911d6883-607c-464e-b46b-8e6141bdd2de]'}), log id: 1438bef 2017-10-01 14:55:53,245-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVDSCommand] (DefaultQuartzScheduler4) [4e42cf4a] FINISH, FullListVDSCommand, return: [{acpiEnable=true, emulatedMachine=pc-i440fx-rhel7.3.0, afterMigrationStatus=, pid=89465, memGuaranteedSize=682, transparentHugePages=true, timeOffset=-14399, cpuType=SandyBridge, smp=1, guestNumaNodes=[Ljava.lang.Object;@691007bf, smartcardEnable=false, custom={device_af719532-f1f7-4c86-9b33-bb40c117eb3cdevice_b9298247-8996-4bc5-b351-ce32dfd4d32e=VmDevice:{id='VmDeviceId:{deviceId='b9298247-8996-4bc5-b351-ce32dfd4d32e', vmId='911d6883-607c-464e-b46b-8e6141bdd2de'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=2}', managed='false', plugged='true', readOnly='false', deviceAlias='channel1', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_af719532-f1f7-4c86-9b33-bb40c117eb3cdevice_b9298247-8996-4bc5-b351-ce32dfd4d32edevice_bd14853c-8a9e-4beb-b8f9-134b3b902c6cdevice_9d2ab0a4-a3b2-4d30-8847-67b3d57ec847=VmDevice:{id='VmDeviceId:{deviceId='9d2ab0a4-a3b2-4d30-8847-67b3d57ec847', vmId='911d6883-607c-464e-b46b-8e6141bdd2de'}', device='ide', type='CONTROLLER', bootOrder='0', specParams='[]', address='{slot=0x01, bus=0x00, domain=0x0000, type=pci, function=0x1}', managed='false', plugged='true', readOnly='false', deviceAlias='ide', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_af719532-f1f7-4c86-9b33-bb40c117eb3cdevice_b9298247-8996-4bc5-b351-ce32dfd4d32edevice_bd14853c-8a9e-4beb-b8f9-134b3b902c6c=VmDevice:{id='VmDeviceId:{deviceId='bd14853c-8a9e-4beb-b8f9-134b3b902c6c', vmId='911d6883-607c-464e-b46b-8e6141bdd2de'}', device='spicevmc', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=3}', managed='false', plugged='true', readOnly='false', deviceAlias='channel2', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_af719532-f1f7-4c86-9b33-bb40c117eb3c=VmDevice:{id='VmDeviceId:{deviceId='af719532-f1f7-4c86-9b33-bb40c117eb3c', vmId='911d6883-607c-464e-b46b-8e6141bdd2de'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=1}', managed='false', plugged='true', readOnly='false', deviceAlias='channel0', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}}, vmType=kvm, memSize=1024, smpCoresPerSocket=1, vmName=hdb, nice=0, status=Up, maxMemSize=4096, bootMenuEnable=false, vmId=911d6883-607c-464e-b46b-8e6141bdd2de, smpThreadsPerCore=1, guestDiskMapping={}, kvmEnable=true, displayNetwork=yorku, devices=[Ljava.lang.Object;@766d1602, display=qxl, maxVCpus=16, clientIp=, statusTime=5251456940, maxMemSlots=16}], log id: 1438bef 2017-10-01 14:55:53,248-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler4) [4e42cf4a] Received a memballoon Device without an address when processing VM 911d6883-607c-464e-b46b-8e6141bdd2de devices, skipping device: {device=memballoon, specParams={model=none}, type=balloon, target=1048576, alias=balloon0} 2017-10-01 14:55:53,248-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler4) [4e42cf4a] Received a spice Device without an address when processing VM 911d6883-607c-464e-b46b-8e6141bdd2de devices, skipping device: {device=spice, specParams={fileTransferEnable=true, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,ssmartcard,susbredir, copyPasteEnable=true, displayIp=130.63.94.37, displayNetwork=yorku}, type=graphics, deviceId=62ba37d1-71cb-4c67-ac3c-bab7f7cf57b2, tlsPort=5909} 2017-10-01 14:55:54,883-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler7) [3ac21e6] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 14:55:56,406-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmsStatisticsFetcher] (DefaultQuartzScheduler6) [5bcc9086] Fetched 9 VMs from VDS '45f8b331-842e-48e7-9df8-56adddb93836' 2017-10-01 14:55:56,406-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler6) [5bcc9086] VM '899e3a1c-dcc4-4426-9dd4-0f1e8b94a2b8'(ipa) moved from 'NotResponding' --> 'Up' 2017-10-01 14:55:56,406-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler6) [5bcc9086] VM '77766cb4-0625-4432-a16f-def5e702102a'(labtest1) moved from 'NotResponding' --> 'Up' 2017-10-01 14:55:56,407-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler6) [5bcc9086] VM '8391f9a7-7fe7-4025-8fba-6273ae587a52' was reported as Down on VDS '45f8b331-842e-48e7-9df8-56adddb93836'(virt1) 2017-10-01 14:55:56,407-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (DefaultQuartzScheduler6) [5bcc9086] START, DestroyVDSCommand(HostName = virt1, DestroyVmVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vmId='8391f9a7-7fe7-4025-8fba-6273ae587a52', force='false', secondsToWait='0', gracefully='false', reason='', ignoreNoVm='true'}), log id: 6c85f537 2017-10-01 14:55:56,494-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (DefaultQuartzScheduler6) [5bcc9086] FINISH, DestroyVDSCommand, log id: 6c85f537 2017-10-01 14:55:56,494-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler6) [5bcc9086] VM '8391f9a7-7fe7-4025-8fba-6273ae587a52'(archive) was unexpectedly detected as 'Down' on VDS '45f8b331-842e-48e7-9df8-56adddb93836'(virt1) (expected on '5ae29ed2-9c0f-4754-8df3-b43d14726846') 2017-10-01 14:55:56,494-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler6) [5bcc9086] VM '59287796-c634-4f3c-94b1-2c9699986637'(vpn1) moved from 'NotResponding' --> 'Up' 2017-10-01 14:56:04,047-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-11) [] VM 'aa487207-7ff4-465a-9d9b-2a103d50dc77'(ftp1) was unexpectedly detected as 'Up' on VDS 'a0ccfd11-3402-4eba-a119-4987561b738e'(virt2) (expected on '45f8b331-842e-48e7-9df8-56adddb93836') 2017-10-01 14:56:04,055-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ForkJoinPool-1-worker-11) [] EVENT_ID: VM_MIGRATION_DONE(63), Correlation ID: 4db23c36, Job ID: fada4f20-3260-460c-879a-f1b7807047c8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Migration completed (VM: ftp1, Source: virt1, Destination: virt2, Duration: 51 seconds, Total: 51 seconds, Actual downtime: 63ms) 2017-10-01 14:56:04,059-04 INFO [org.ovirt.engine.core.bll.MigrateVmCommand] (ForkJoinPool-1-worker-11) [] Lock freed to object 'EngineLock:{exclusiveLocks='[aa487207-7ff4-465a-9d9b-2a103d50dc77=VM]', sharedLocks=''}' 2017-10-01 14:56:04,060-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVDSCommand] (ForkJoinPool-1-worker-11) [] START, FullListVDSCommand(HostName = virt2, FullListVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vmIds='[aa487207-7ff4-465a-9d9b-2a103d50dc77]'}), log id: 7a0e6a2e 2017-10-01 14:56:05,077-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVDSCommand] (ForkJoinPool-1-worker-11) [] FINISH, FullListVDSCommand, return: [{acpiEnable=true, emulatedMachine=pc-i440fx-rhel7.3.0, afterMigrationStatus=, pid=36911, memGuaranteedSize=1365, transparentHugePages=true, timeOffset=-14400, cpuType=SandyBridge, smp=1, guestNumaNodes=[Ljava.lang.Object;@6977dc14, smartcardEnable=false, custom={device_79534ec7-8fe5-41d6-95e9-08eff595ecd0device_6b3956c7-76a7-45c5-b4c7-5910c316ae78device_0eec815c-24e1-45fc-921c-0b1b0f7d214b=VmDevice:{id='VmDeviceId:{deviceId='0eec815c-24e1-45fc-921c-0b1b0f7d214b', vmId='aa487207-7ff4-465a-9d9b-2a103d50dc77'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=2}', managed='false', plugged='true', readOnly='false', deviceAlias='channel1', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_79534ec7-8fe5-41d6-95e9-08eff595ecd0=VmDevice:{id='VmDeviceId:{deviceId='79534ec7-8fe5-41d6-95e9-08eff595ecd0', vmId='aa487207-7ff4-465a-9d9b-2a103d50dc77'}', device='ide', type='CONTROLLER', bootOrder='0', specParams='[]', address='{slot=0x01, bus=0x00, domain=0x0000, type=pci, function=0x1}', managed='false', plugged='true', readOnly='false', deviceAlias='ide', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_79534ec7-8fe5-41d6-95e9-08eff595ecd0device_6b3956c7-76a7-45c5-b4c7-5910c316ae78=VmDevice:{id='VmDeviceId:{deviceId='6b3956c7-76a7-45c5-b4c7-5910c316ae78', vmId='aa487207-7ff4-465a-9d9b-2a103d50dc77'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=1}', managed='false', plugged='true', readOnly='false', deviceAlias='channel0', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_79534ec7-8fe5-41d6-95e9-08eff595ecd0device_6b3956c7-76a7-45c5-b4c7-5910c316ae78device_0eec815c-24e1-45fc-921c-0b1b0f7d214bdevice_c5ad6a7a-753e-4edd-9d9a-4f1047858be1=VmDevice:{id='VmDeviceId:{deviceId='c5ad6a7a-753e-4edd-9d9a-4f1047858be1', vmId='aa487207-7ff4-465a-9d9b-2a103d50dc77'}', device='spicevmc', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=3}', managed='false', plugged='true', readOnly='false', deviceAlias='channel2', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}}, vmType=kvm, memSize=2048, smpCoresPerSocket=1, vmName=ftp1, nice=0, status=Up, maxMemSize=8192, bootMenuEnable=false, vmId=aa487207-7ff4-465a-9d9b-2a103d50dc77, smpThreadsPerCore=1, guestDiskMapping={}, kvmEnable=true, displayNetwork=yorku, devices=[Ljava.lang.Object;@6a9c680b, display=qxl, maxVCpus=16, clientIp=, statusTime=4720847290, maxMemSlots=16}], log id: 7a0e6a2e 2017-10-01 14:56:05,078-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (ForkJoinPool-1-worker-11) [] Received a spice Device without an address when processing VM aa487207-7ff4-465a-9d9b-2a103d50dc77 devices, skipping device: {device=spice, specParams={fileTransferEnable=true, displayNetwork=yorku, copyPasteEnable=true, displayIp=130.63.94.35, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,ssmartcard,susbredir}, type=graphics, deviceId=00000024-0024-0024-0024-0000000001c3, tlsPort=5904} 2017-10-01 14:56:07,282-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler6) [5e6e9628] VM 'bfe9a524-2238-4bab-a91f-5dc6ed270896'(pcmode) was unexpectedly detected as 'MigratingTo' on VDS 'a0ccfd11-3402-4eba-a119-4987561b738e'(virt2) (expected on '45f8b331-842e-48e7-9df8-56adddb93836') 2017-10-01 14:56:07,282-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler6) [5e6e9628] VM 'bfe9a524-2238-4bab-a91f-5dc6ed270896' is migrating to VDS 'a0ccfd11-3402-4eba-a119-4987561b738e'(virt2) ignoring it in the refresh until migration is done 2017-10-01 14:56:09,263-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler9) [257f4545] VM '07331c78-6e09-41b4-9b29-edda32f46351'(samba) moved from 'NotResponding' --> 'Up' 2017-10-01 14:56:09,263-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler9) [257f4545] VM 'dc66c051-24b7-4cb8-99eb-db398d396ef1'(samba1) moved from 'NotResponding' --> 'Up' 2017-10-01 14:56:09,264-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler9) [257f4545] VM '1c7e966f-29e8-4788-b64c-86e52f415061'(udb) was unexpectedly detected as 'MigratingTo' on VDS '5ae29ed2-9c0f-4754-8df3-b43d14726846'(virt4) (expected on 'e1e71e9d-a0cd-47db-96c6-f724f6cea736') 2017-10-01 14:56:09,264-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler9) [257f4545] VM '1c7e966f-29e8-4788-b64c-86e52f415061' is migrating to VDS '5ae29ed2-9c0f-4754-8df3-b43d14726846'(virt4) ignoring it in the refresh until migration is done 2017-10-01 14:56:09,264-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler9) [257f4545] VM '9ea2821e-557f-45cb-9872-1e6c5aeaf246'(rs) was unexpectedly detected as 'Up' on VDS '5ae29ed2-9c0f-4754-8df3-b43d14726846'(virt4) (expected on 'e1e71e9d-a0cd-47db-96c6-f724f6cea736') 2017-10-01 14:56:09,272-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [257f4545] EVENT_ID: VM_MIGRATION_DONE(63), Correlation ID: 1a031d8d, Job ID: 1d6ef5c3-aab0-40f3-97f1-52aee6279df1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Migration completed (VM: rs, Source: virt3, Destination: virt4, Duration: 59 seconds, Total: 59 seconds, Actual downtime: 38ms) 2017-10-01 14:56:09,275-04 INFO [org.ovirt.engine.core.bll.MigrateVmCommand] (DefaultQuartzScheduler9) [257f4545] Lock freed to object 'EngineLock:{exclusiveLocks='[9ea2821e-557f-45cb-9872-1e6c5aeaf246=VM]', sharedLocks=''}' 2017-10-01 14:56:09,276-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVDSCommand] (DefaultQuartzScheduler9) [257f4545] START, FullListVDSCommand(HostName = virt4, FullListVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vmIds='[8391f9a7-7fe7-4025-8fba-6273ae587a52]'}), log id: 7f9295bd 2017-10-01 14:56:09,284-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler7) [4e602814] VM '1734fed5-fc97-4714-bb63-55e8f4eb6d34'(wiki) moved from 'NotResponding' --> 'Up' 2017-10-01 14:56:09,285-04 WARN [org.ovirt.engine.core.vdsbroker.monitoring.VmsMonitoring] (DefaultQuartzScheduler7) [4e602814] skipping VM '1c7e966f-29e8-4788-b64c-86e52f415061' from this monitoring cycle - the VM data has changed since fetching the data 2017-10-01 14:56:09,285-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler7) [4e602814] VM 'cc1dbe93-bb80-451e-8b3b-c9cc5e90432f'(license) moved from 'NotResponding' --> 'Up' 2017-10-01 14:56:09,285-04 WARN [org.ovirt.engine.core.vdsbroker.monitoring.VmsMonitoring] (DefaultQuartzScheduler7) [4e602814] skipping VM '9ea2821e-557f-45cb-9872-1e6c5aeaf246' from this monitoring cycle - the VM data has changed since fetching the data 2017-10-01 14:56:09,285-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler7) [4e602814] VM '64734fca-4d6c-443c-82d1-abf7cf106601'(webdav) moved from 'NotResponding' --> 'Up' 2017-10-01 14:56:10,099-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVDSCommand] (DefaultQuartzScheduler9) [257f4545] FINISH, FullListVDSCommand, return: [{acpiEnable=true, emulatedMachine=pc-i440fx-rhel7.3.0, afterMigrationStatus=, pid=89735, memGuaranteedSize=682, transparentHugePages=true, timeOffset=-14400, cpuType=SandyBridge, smp=1, guestNumaNodes=[Ljava.lang.Object;@3a35cbd6, smartcardEnable=false, custom={device_0b1ea32d-69bc-4031-9820-74effd839c96=VmDevice:{id='VmDeviceId:{deviceId='0b1ea32d-69bc-4031-9820-74effd839c96', vmId='8391f9a7-7fe7-4025-8fba-6273ae587a52'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=1}', managed='false', plugged='true', readOnly='false', deviceAlias='channel0', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_0b1ea32d-69bc-4031-9820-74effd839c96device_0f40e9cc-5a3e-4978-a62d-1638f44ad70ddevice_5c938868-4713-47ec-8a2a-c8dbdb6aadc2device_05451551-412a-49b4-8988-35d60926d78d=VmDevice:{id='VmDeviceId:{deviceId='05451551-412a-49b4-8988-35d60926d78d', vmId='8391f9a7-7fe7-4025-8fba-6273ae587a52'}', device='ide', type='CONTROLLER', bootOrder='0', specParams='[]', address='{slot=0x01, bus=0x00, domain=0x0000, type=pci, function=0x1}', managed='false', plugged='true', readOnly='false', deviceAlias='ide', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_0b1ea32d-69bc-4031-9820-74effd839c96device_0f40e9cc-5a3e-4978-a62d-1638f44ad70d=VmDevice:{id='VmDeviceId:{deviceId='0f40e9cc-5a3e-4978-a62d-1638f44ad70d', vmId='8391f9a7-7fe7-4025-8fba-6273ae587a52'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=2}', managed='false', plugged='true', readOnly='false', deviceAlias='channel1', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_0b1ea32d-69bc-4031-9820-74effd839c96device_0f40e9cc-5a3e-4978-a62d-1638f44ad70ddevice_5c938868-4713-47ec-8a2a-c8dbdb6aadc2=VmDevice:{id='VmDeviceId:{deviceId='5c938868-4713-47ec-8a2a-c8dbdb6aadc2', vmId='8391f9a7-7fe7-4025-8fba-6273ae587a52'}', device='spicevmc', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=3}', managed='false', plugged='true', readOnly='false', deviceAlias='channel2', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}}, vmType=kvm, memSize=1024, smpCoresPerSocket=1, vmName=archive, nice=0, status=Up, maxMemSize=4096, bootMenuEnable=false, vmId=8391f9a7-7fe7-4025-8fba-6273ae587a52, smpThreadsPerCore=1, guestDiskMapping={}, maxMemSlots=16, kvmEnable=true, displayNetwork=yorku, devices=[Ljava.lang.Object;@1f09941, maxVCpus=16, clientIp=, statusTime=5251473950, display=qxl}], log id: 7f9295bd 2017-10-01 14:56:10,100-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler9) [257f4545] Received a memballoon Device without an address when processing VM 8391f9a7-7fe7-4025-8fba-6273ae587a52 devices, skipping device: {device=memballoon, specParams={model=none}, type=balloon, target=1048576, alias=balloon0} 2017-10-01 14:56:10,100-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler9) [257f4545] Received a spice Device without an address when processing VM 8391f9a7-7fe7-4025-8fba-6273ae587a52 devices, skipping device: {device=spice, specParams={fileTransferEnable=true, displayNetwork=yorku, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,ssmartcard,susbredir, displayIp=130.63.94.37, copyPasteEnable=true}, type=graphics, deviceId=c520628b-2ea8-4b60-b6c6-590abc0e7ef8, tlsPort=5911} 2017-10-01 14:56:11,795-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmsStatisticsFetcher] (DefaultQuartzScheduler10) [4faee34c] Fetched 8 VMs from VDS '45f8b331-842e-48e7-9df8-56adddb93836' 2017-10-01 14:56:11,799-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler10) [4faee34c] VM 'aa487207-7ff4-465a-9d9b-2a103d50dc77' was reported as Down on VDS '45f8b331-842e-48e7-9df8-56adddb93836'(virt1) 2017-10-01 14:56:11,799-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (DefaultQuartzScheduler10) [4faee34c] START, DestroyVDSCommand(HostName = virt1, DestroyVmVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vmId='aa487207-7ff4-465a-9d9b-2a103d50dc77', force='false', secondsToWait='0', gracefully='false', reason='', ignoreNoVm='true'}), log id: 2c6b2eb8 2017-10-01 14:56:11,811-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (DefaultQuartzScheduler10) [4faee34c] FINISH, DestroyVDSCommand, log id: 2c6b2eb8 2017-10-01 14:56:11,811-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler10) [4faee34c] VM 'aa487207-7ff4-465a-9d9b-2a103d50dc77'(ftp1) was unexpectedly detected as 'Down' on VDS '45f8b331-842e-48e7-9df8-56adddb93836'(virt1) (expected on 'a0ccfd11-3402-4eba-a119-4987561b738e') 2017-10-01 14:56:22,297-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmsStatisticsFetcher] (DefaultQuartzScheduler6) [54a1e249] Fetched 9 VMs from VDS 'a0ccfd11-3402-4eba-a119-4987561b738e' 2017-10-01 14:56:22,298-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler6) [54a1e249] VM 'bfe9a524-2238-4bab-a91f-5dc6ed270896'(pcmode) was unexpectedly detected as 'MigratingTo' on VDS 'a0ccfd11-3402-4eba-a119-4987561b738e'(virt2) (expected on '45f8b331-842e-48e7-9df8-56adddb93836') 2017-10-01 14:56:22,298-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler6) [54a1e249] VM 'bfe9a524-2238-4bab-a91f-5dc6ed270896' is migrating to VDS 'a0ccfd11-3402-4eba-a119-4987561b738e'(virt2) ignoring it in the refresh until migration is done 2017-10-01 14:56:22,299-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler6) [54a1e249] VM '1325cebf-834e-4571-a3e1-82beb0021ffe'(dns1) was unexpectedly detected as 'MigratingTo' on VDS 'a0ccfd11-3402-4eba-a119-4987561b738e'(virt2) (expected on '45f8b331-842e-48e7-9df8-56adddb93836') 2017-10-01 14:56:22,299-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler6) [54a1e249] VM '1325cebf-834e-4571-a3e1-82beb0021ffe' is migrating to VDS 'a0ccfd11-3402-4eba-a119-4987561b738e'(virt2) ignoring it in the refresh until migration is done 2017-10-01 14:56:25,119-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler6) [3fe6c4f] VM '1c7e966f-29e8-4788-b64c-86e52f415061' was reported as Down on VDS 'e1e71e9d-a0cd-47db-96c6-f724f6cea736'(virt3) 2017-10-01 14:56:25,119-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (DefaultQuartzScheduler6) [3fe6c4f] START, DestroyVDSCommand(HostName = virt3, DestroyVmVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vmId='1c7e966f-29e8-4788-b64c-86e52f415061', force='false', secondsToWait='0', gracefully='false', reason='', ignoreNoVm='true'}), log id: 32c168bd 2017-10-01 14:56:25,131-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVDSCommand] (DefaultQuartzScheduler1) [74d541cb] START, FullListVDSCommand(HostName = virt4, FullListVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vmIds='[9ea2821e-557f-45cb-9872-1e6c5aeaf246]'}), log id: 30f53767 2017-10-01 14:56:25,150-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (DefaultQuartzScheduler6) [3fe6c4f] FINISH, DestroyVDSCommand, log id: 32c168bd 2017-10-01 14:56:25,150-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler6) [3fe6c4f] VM '1c7e966f-29e8-4788-b64c-86e52f415061'(udb) moved from 'MigratingFrom' --> 'Down' 2017-10-01 14:56:25,151-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler6) [3fe6c4f] Handing over VM '1c7e966f-29e8-4788-b64c-86e52f415061'(udb) to Host '5ae29ed2-9c0f-4754-8df3-b43d14726846'. Setting VM to status 'MigratingTo' 2017-10-01 14:56:25,151-04 WARN [org.ovirt.engine.core.vdsbroker.monitoring.VmsMonitoring] (DefaultQuartzScheduler6) [3fe6c4f] skipping VM '9ea2821e-557f-45cb-9872-1e6c5aeaf246' from this monitoring cycle - the VM data has changed since fetching the data 2017-10-01 14:56:25,330-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVDSCommand] (DefaultQuartzScheduler1) [74d541cb] FINISH, FullListVDSCommand, return: [{acpiEnable=true, emulatedMachine=pc-i440fx-rhel7.3.0, afterMigrationStatus=, pid=90060, memGuaranteedSize=682, transparentHugePages=true, timeOffset=0, cpuType=SandyBridge, smp=1, guestNumaNodes=[Ljava.lang.Object;@291dbf2f, smartcardEnable=false, custom={device_9d97bcb6-afab-4883-bd18-b4fa17c38646device_7f13f492-45bf-462e-bc1f-9df15ab83d0adevice_8e0a9689-cb82-4ae0-8dde-516ea29821b0device_0ad46c28-4593-4387-8c7a-fcc41bdb7065=VmDevice:{id='VmDeviceId:{deviceId='0ad46c28-4593-4387-8c7a-fcc41bdb7065', vmId='9ea2821e-557f-45cb-9872-1e6c5aeaf246'}', device='ide', type='CONTROLLER', bootOrder='0', specParams='[]', address='{slot=0x01, bus=0x00, domain=0x0000, type=pci, function=0x1}', managed='false', plugged='true', readOnly='false', deviceAlias='ide', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_9d97bcb6-afab-4883-bd18-b4fa17c38646device_7f13f492-45bf-462e-bc1f-9df15ab83d0a=VmDevice:{id='VmDeviceId:{deviceId='7f13f492-45bf-462e-bc1f-9df15ab83d0a', vmId='9ea2821e-557f-45cb-9872-1e6c5aeaf246'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=2}', managed='false', plugged='true', readOnly='false', deviceAlias='channel1', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_9d97bcb6-afab-4883-bd18-b4fa17c38646=VmDevice:{id='VmDeviceId:{deviceId='9d97bcb6-afab-4883-bd18-b4fa17c38646', vmId='9ea2821e-557f-45cb-9872-1e6c5aeaf246'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=1}', managed='false', plugged='true', readOnly='false', deviceAlias='channel0', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_9d97bcb6-afab-4883-bd18-b4fa17c38646device_7f13f492-45bf-462e-bc1f-9df15ab83d0adevice_8e0a9689-cb82-4ae0-8dde-516ea29821b0=VmDevice:{id='VmDeviceId:{deviceId='8e0a9689-cb82-4ae0-8dde-516ea29821b0', vmId='9ea2821e-557f-45cb-9872-1e6c5aeaf246'}', device='spicevmc', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=3}', managed='false', plugged='true', readOnly='false', deviceAlias='channel2', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}}, vmType=kvm, memSize=1024, smpCoresPerSocket=1, vmName=rs, nice=0, status=Up, maxMemSize=4096, bootMenuEnable=false, vmId=9ea2821e-557f-45cb-9872-1e6c5aeaf246, smpThreadsPerCore=1, guestDiskMapping={}, kvmEnable=true, displayNetwork=yorku, devices=[Ljava.lang.Object;@3b2a5f4a, display=qxl, maxVCpus=16, clientIp=, statusTime=5251489810, maxMemSlots=16}], log id: 30f53767 2017-10-01 14:56:25,334-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler1) [74d541cb] Received a memballoon Device without an address when processing VM 9ea2821e-557f-45cb-9872-1e6c5aeaf246 devices, skipping device: {device=memballoon, specParams={model=none}, type=balloon, target=1048576, alias=balloon0} 2017-10-01 14:56:25,334-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler1) [74d541cb] Received a spice Device without an address when processing VM 9ea2821e-557f-45cb-9872-1e6c5aeaf246 devices, skipping device: {device=spice, specParams={fileTransferEnable=true, displayNetwork=yorku, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,ssmartcard,susbredir, displayIp=130.63.94.37, copyPasteEnable=true}, type=graphics, deviceId=58c8cadc-6a62-488c-85d2-dc06c41907f3, tlsPort=5912} 2017-10-01 14:56:26,830-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmsStatisticsFetcher] (DefaultQuartzScheduler7) [7aff6ff2] Fetched 7 VMs from VDS '45f8b331-842e-48e7-9df8-56adddb93836' 2017-10-01 14:56:33,577-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-4) [] VM 'bfe9a524-2238-4bab-a91f-5dc6ed270896'(pcmode) was unexpectedly detected as 'Up' on VDS 'a0ccfd11-3402-4eba-a119-4987561b738e'(virt2) (expected on '45f8b331-842e-48e7-9df8-56adddb93836') 2017-10-01 14:56:33,586-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ForkJoinPool-1-worker-4) [] EVENT_ID: VM_MIGRATION_DONE(63), Correlation ID: 2346e2d3, Job ID: e47484f4-ac65-4022-8f7c-4f9a883bdc9d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Migration completed (VM: pcmode, Source: virt1, Destination: virt2, Duration: 1 minute 19 seconds, Total: 1 minute 19 seconds, Actual downtime: 45ms) 2017-10-01 14:56:33,589-04 INFO [org.ovirt.engine.core.bll.MigrateVmCommand] (ForkJoinPool-1-worker-4) [] Lock freed to object 'EngineLock:{exclusiveLocks='[bfe9a524-2238-4bab-a91f-5dc6ed270896=VM]', sharedLocks=''}' 2017-10-01 14:56:33,590-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVDSCommand] (ForkJoinPool-1-worker-4) [] START, FullListVDSCommand(HostName = virt2, FullListVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vmIds='[bfe9a524-2238-4bab-a91f-5dc6ed270896]'}), log id: 39f852f3 2017-10-01 14:56:34,606-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVDSCommand] (ForkJoinPool-1-worker-4) [] FINISH, FullListVDSCommand, return: [{acpiEnable=true, emulatedMachine=pc-i440fx-rhel7.3.0, afterMigrationStatus=, pid=37182, memGuaranteedSize=4096, transparentHugePages=true, timeOffset=-14399, cpuType=SandyBridge, smp=1, guestNumaNodes=[Ljava.lang.Object;@5c65e13d, smartcardEnable=false, custom={device_6d36367b-1fec-4258-a874-5421537d982b=VmDevice:{id='VmDeviceId:{deviceId='6d36367b-1fec-4258-a874-5421537d982b', vmId='bfe9a524-2238-4bab-a91f-5dc6ed270896'}', device='ide', type='CONTROLLER', bootOrder='0', specParams='[]', address='{slot=0x01, bus=0x00, domain=0x0000, type=pci, function=0x1}', managed='false', plugged='true', readOnly='false', deviceAlias='ide', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_6d36367b-1fec-4258-a874-5421537d982bdevice_4166ecf0-4947-40b1-a1e5-c4255cc01ec2device_774ecbec-6de7-407d-89ee-800de5eca602device_bcfd6fef-9c94-471a-a75e-321be70c517f=VmDevice:{id='VmDeviceId:{deviceId='bcfd6fef-9c94-471a-a75e-321be70c517f', vmId='bfe9a524-2238-4bab-a91f-5dc6ed270896'}', device='spicevmc', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=3}', managed='false', plugged='true', readOnly='false', deviceAlias='channel2', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_6d36367b-1fec-4258-a874-5421537d982bdevice_4166ecf0-4947-40b1-a1e5-c4255cc01ec2=VmDevice:{id='VmDeviceId:{deviceId='4166ecf0-4947-40b1-a1e5-c4255cc01ec2', vmId='bfe9a524-2238-4bab-a91f-5dc6ed270896'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=1}', managed='false', plugged='true', readOnly='false', deviceAlias='channel0', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_6d36367b-1fec-4258-a874-5421537d982bdevice_4166ecf0-4947-40b1-a1e5-c4255cc01ec2device_774ecbec-6de7-407d-89ee-800de5eca602=VmDevice:{id='VmDeviceId:{deviceId='774ecbec-6de7-407d-89ee-800de5eca602', vmId='bfe9a524-2238-4bab-a91f-5dc6ed270896'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=2}', managed='false', plugged='true', readOnly='false', deviceAlias='channel1', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}}, vmType=kvm, memSize=4096, smpCoresPerSocket=1, vmName=pcmode, nice=0, status=Up, maxMemSize=16384, bootMenuEnable=false, vmId=bfe9a524-2238-4bab-a91f-5dc6ed270896, smpThreadsPerCore=1, guestDiskMapping={}, kvmEnable=true, displayNetwork=yorku, devices=[Ljava.lang.Object;@28a9f77c, display=qxl, maxVCpus=16, clientIp=, statusTime=4720876820, maxMemSlots=16}], log id: 39f852f3 2017-10-01 14:56:34,609-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (ForkJoinPool-1-worker-4) [] Received a memballoon Device without an address when processing VM bfe9a524-2238-4bab-a91f-5dc6ed270896 devices, skipping device: {device=memballoon, specParams={model=none}, type=balloon, target=4194304, alias=balloon0} 2017-10-01 14:56:34,609-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (ForkJoinPool-1-worker-4) [] Received a spice Device without an address when processing VM bfe9a524-2238-4bab-a91f-5dc6ed270896 devices, skipping device: {device=spice, specParams={fileTransferEnable=true, displayNetwork=yorku, copyPasteEnable=true, displayIp=130.63.94.35, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,ssmartcard,susbredir}, type=graphics, deviceId=ca0f5b5f-d51f-4838-8042-a1e14722e179, tlsPort=5906} 2017-10-01 14:56:37,326-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler5) [111b016b] VM '1325cebf-834e-4571-a3e1-82beb0021ffe'(dns1) was unexpectedly detected as 'MigratingTo' on VDS 'a0ccfd11-3402-4eba-a119-4987561b738e'(virt2) (expected on '45f8b331-842e-48e7-9df8-56adddb93836') 2017-10-01 14:56:37,326-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler5) [111b016b] VM '1325cebf-834e-4571-a3e1-82beb0021ffe' is migrating to VDS 'a0ccfd11-3402-4eba-a119-4987561b738e'(virt2) ignoring it in the refresh until migration is done 2017-10-01 14:56:40,354-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmsStatisticsFetcher] (DefaultQuartzScheduler4) [186f525b] Fetched 5 VMs from VDS 'e1e71e9d-a0cd-47db-96c6-f724f6cea736' 2017-10-01 14:56:40,356-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler4) [186f525b] VM '9ea2821e-557f-45cb-9872-1e6c5aeaf246' was reported as Down on VDS 'e1e71e9d-a0cd-47db-96c6-f724f6cea736'(virt3) 2017-10-01 14:56:40,356-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (DefaultQuartzScheduler4) [186f525b] START, DestroyVDSCommand(HostName = virt3, DestroyVmVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vmId='9ea2821e-557f-45cb-9872-1e6c5aeaf246', force='false', secondsToWait='0', gracefully='false', reason='', ignoreNoVm='true'}), log id: 449b4ad8 2017-10-01 14:56:40,361-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler5) [5819598b] VM '1c7e966f-29e8-4788-b64c-86e52f415061'(udb) moved from 'MigratingTo' --> 'Up' 2017-10-01 14:56:40,376-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [5819598b] EVENT_ID: VM_MIGRATION_DONE(63), Correlation ID: 58940caa, Job ID: 37f56cd4-078d-4bec-bce5-ab1ca2c5cda2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Migration completed (VM: udb, Source: virt3, Destination: virt4, Duration: 1 minute 30 seconds, Total: 1 minute 30 seconds, Actual downtime: 52ms) 2017-10-01 14:56:40,380-04 INFO [org.ovirt.engine.core.bll.MigrateVmCommand] (DefaultQuartzScheduler5) [5819598b] Lock freed to object 'EngineLock:{exclusiveLocks='[1c7e966f-29e8-4788-b64c-86e52f415061=VM]', sharedLocks=''}' 2017-10-01 14:56:40,381-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVDSCommand] (DefaultQuartzScheduler5) [5819598b] START, FullListVDSCommand(HostName = virt4, FullListVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vmIds='[1c7e966f-29e8-4788-b64c-86e52f415061]'}), log id: 20ebca60 2017-10-01 14:56:40,388-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (DefaultQuartzScheduler4) [186f525b] FINISH, DestroyVDSCommand, log id: 449b4ad8 2017-10-01 14:56:40,388-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler4) [186f525b] VM '9ea2821e-557f-45cb-9872-1e6c5aeaf246'(rs) was unexpectedly detected as 'Down' on VDS 'e1e71e9d-a0cd-47db-96c6-f724f6cea736'(virt3) (expected on '5ae29ed2-9c0f-4754-8df3-b43d14726846') 2017-10-01 14:56:40,475-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVDSCommand] (DefaultQuartzScheduler5) [5819598b] FINISH, FullListVDSCommand, return: [{acpiEnable=true, emulatedMachine=pc-i440fx-rhel7.3.0, afterMigrationStatus=, pid=89396, memGuaranteedSize=4096, transparentHugePages=true, timeOffset=-28800, cpuType=SandyBridge, smp=2, guestNumaNodes=[Ljava.lang.Object;@44f7a669, smartcardEnable=false, custom={device_430b6b09-cbd4-4db7-92d1-46c48f7d59e0device_9a357786-4848-4acb-964e-a021ac1f5f8ddevice_8aa12d8d-e699-4709-9a3a-d472b47eed34=VmDevice:{id='VmDeviceId:{deviceId='8aa12d8d-e699-4709-9a3a-d472b47eed34', vmId='1c7e966f-29e8-4788-b64c-86e52f415061'}', device='spicevmc', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=3}', managed='false', plugged='true', readOnly='false', deviceAlias='channel2', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_430b6b09-cbd4-4db7-92d1-46c48f7d59e0device_9a357786-4848-4acb-964e-a021ac1f5f8ddevice_8aa12d8d-e699-4709-9a3a-d472b47eed34device_55bd64e8-23e3-4a64-8285-aa16f8b23aed=VmDevice:{id='VmDeviceId:{deviceId='55bd64e8-23e3-4a64-8285-aa16f8b23aed', vmId='1c7e966f-29e8-4788-b64c-86e52f415061'}', device='ide', type='CONTROLLER', bootOrder='0', specParams='[]', address='{slot=0x01, bus=0x00, domain=0x0000, type=pci, function=0x1}', managed='false', plugged='true', readOnly='false', deviceAlias='ide', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_430b6b09-cbd4-4db7-92d1-46c48f7d59e0=VmDevice:{id='VmDeviceId:{deviceId='430b6b09-cbd4-4db7-92d1-46c48f7d59e0', vmId='1c7e966f-29e8-4788-b64c-86e52f415061'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=1}', managed='false', plugged='true', readOnly='false', deviceAlias='channel0', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_430b6b09-cbd4-4db7-92d1-46c48f7d59e0device_9a357786-4848-4acb-964e-a021ac1f5f8d=VmDevice:{id='VmDeviceId:{deviceId='9a357786-4848-4acb-964e-a021ac1f5f8d', vmId='1c7e966f-29e8-4788-b64c-86e52f415061'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=2}', managed='false', plugged='true', readOnly='false', deviceAlias='channel1', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}}, vmType=kvm, memSize=4096, smpCoresPerSocket=1, vmName=udb, nice=0, status=Up, maxMemSize=16384, bootMenuEnable=false, vmId=1c7e966f-29e8-4788-b64c-86e52f415061, smpThreadsPerCore=1, guestDiskMapping={}, kvmEnable=true, displayNetwork=yorku, devices=[Ljava.lang.Object;@a8cb8b0, display=qxl, maxVCpus=16, clientIp=, statusTime=5251505060, maxMemSlots=16}], log id: 20ebca60 2017-10-01 14:56:40,479-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler5) [5819598b] Received a memballoon Device without an address when processing VM 1c7e966f-29e8-4788-b64c-86e52f415061 devices, skipping device: {device=memballoon, specParams={model=none}, type=balloon, target=4194304, alias=balloon0} 2017-10-01 14:56:40,479-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler5) [5819598b] Received a spice Device without an address when processing VM 1c7e966f-29e8-4788-b64c-86e52f415061 devices, skipping device: {device=spice, specParams={fileTransferEnable=true, displayNetwork=yorku, copyPasteEnable=true, displayIp=130.63.94.37, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,ssmartcard,susbredir}, type=graphics, deviceId=05aeb875-80f2-4a28-9070-4975d8dc2b9b, tlsPort=5910} 2017-10-01 14:56:41,851-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler2) [103dc7a4] VM 'bfe9a524-2238-4bab-a91f-5dc6ed270896' was reported as Down on VDS '45f8b331-842e-48e7-9df8-56adddb93836'(virt1) 2017-10-01 14:56:41,851-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (DefaultQuartzScheduler2) [103dc7a4] START, DestroyVDSCommand(HostName = virt1, DestroyVmVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vmId='bfe9a524-2238-4bab-a91f-5dc6ed270896', force='false', secondsToWait='0', gracefully='false', reason='', ignoreNoVm='true'}), log id: b7ae115 2017-10-01 14:56:42,681-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (DefaultQuartzScheduler2) [103dc7a4] FINISH, DestroyVDSCommand, log id: b7ae115 2017-10-01 14:56:42,681-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler2) [103dc7a4] VM 'bfe9a524-2238-4bab-a91f-5dc6ed270896'(pcmode) was unexpectedly detected as 'Down' on VDS '45f8b331-842e-48e7-9df8-56adddb93836'(virt1) (expected on 'a0ccfd11-3402-4eba-a119-4987561b738e') 2017-10-01 14:56:50,306-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-11) [] VM '1325cebf-834e-4571-a3e1-82beb0021ffe'(dns1) was unexpectedly detected as 'Up' on VDS 'a0ccfd11-3402-4eba-a119-4987561b738e'(virt2) (expected on '45f8b331-842e-48e7-9df8-56adddb93836') 2017-10-01 14:56:50,312-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ForkJoinPool-1-worker-11) [] EVENT_ID: VM_MIGRATION_DONE(63), Correlation ID: 325def75, Job ID: f694cca6-07c0-4595-82f7-155e1765e0eb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Migration completed (VM: dns1, Source: virt1, Destination: virt2, Duration: 1 minute 35 seconds, Total: 1 minute 35 seconds, Actual downtime: 47ms) 2017-10-01 14:56:50,315-04 INFO [org.ovirt.engine.core.bll.MigrateVmCommand] (ForkJoinPool-1-worker-11) [] Lock freed to object 'EngineLock:{exclusiveLocks='[1325cebf-834e-4571-a3e1-82beb0021ffe=VM]', sharedLocks=''}' 2017-10-01 14:56:50,315-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVDSCommand] (ForkJoinPool-1-worker-11) [] START, FullListVDSCommand(HostName = virt2, FullListVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vmIds='[1325cebf-834e-4571-a3e1-82beb0021ffe]'}), log id: 703580a8 2017-10-01 14:56:51,332-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVDSCommand] (ForkJoinPool-1-worker-11) [] FINISH, FullListVDSCommand, return: [{acpiEnable=true, emulatedMachine=pc-i440fx-rhel7.3.0, afterMigrationStatus=, pid=37436, memGuaranteedSize=1365, transparentHugePages=true, timeOffset=0, cpuType=SandyBridge, smp=1, guestNumaNodes=[Ljava.lang.Object;@30c86f21, smartcardEnable=false, custom={device_36e42d1d-81f3-4659-b27d-27daaa597ea9=VmDevice:{id='VmDeviceId:{deviceId='36e42d1d-81f3-4659-b27d-27daaa597ea9', vmId='1325cebf-834e-4571-a3e1-82beb0021ffe'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=1}', managed='false', plugged='true', readOnly='false', deviceAlias='channel0', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_36e42d1d-81f3-4659-b27d-27daaa597ea9device_5b63eabc-b66c-429d-92d1-d7ce6db12dc0=VmDevice:{id='VmDeviceId:{deviceId='5b63eabc-b66c-429d-92d1-d7ce6db12dc0', vmId='1325cebf-834e-4571-a3e1-82beb0021ffe'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=2}', managed='false', plugged='true', readOnly='false', deviceAlias='channel1', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_36e42d1d-81f3-4659-b27d-27daaa597ea9device_5b63eabc-b66c-429d-92d1-d7ce6db12dc0device_eadba48d-d976-4810-a602-6762ca8abdbbdevice_3a0eb1cd-ac90-4e07-8b81-2a7743c5b4f2=VmDevice:{id='VmDeviceId:{deviceId='3a0eb1cd-ac90-4e07-8b81-2a7743c5b4f2', vmId='1325cebf-834e-4571-a3e1-82beb0021ffe'}', device='ide', type='CONTROLLER', bootOrder='0', specParams='[]', address='{slot=0x01, bus=0x00, domain=0x0000, type=pci, function=0x1}', managed='false', plugged='true', readOnly='false', deviceAlias='ide', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_36e42d1d-81f3-4659-b27d-27daaa597ea9device_5b63eabc-b66c-429d-92d1-d7ce6db12dc0device_eadba48d-d976-4810-a602-6762ca8abdbb=VmDevice:{id='VmDeviceId:{deviceId='eadba48d-d976-4810-a602-6762ca8abdbb', vmId='1325cebf-834e-4571-a3e1-82beb0021ffe'}', device='spicevmc', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=3}', managed='false', plugged='true', readOnly='false', deviceAlias='channel2', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}}, vmType=kvm, memSize=2048, smpCoresPerSocket=1, vmName=dns1, nice=0, status=Up, maxMemSize=8192, bootMenuEnable=false, vmId=1325cebf-834e-4571-a3e1-82beb0021ffe, smpThreadsPerCore=1, guestDiskMapping={}, kvmEnable=true, displayNetwork=yorku, devices=[Ljava.lang.Object;@38be3895, display=qxl, maxVCpus=16, clientIp=, statusTime=4720893550, maxMemSlots=16}], log id: 703580a8 2017-10-01 14:56:51,332-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (ForkJoinPool-1-worker-11) [] Received a memballoon Device without an address when processing VM 1325cebf-834e-4571-a3e1-82beb0021ffe devices, skipping device: {device=memballoon, specParams={model=none}, type=balloon, target=2097152, alias=balloon0} 2017-10-01 14:56:51,332-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (ForkJoinPool-1-worker-11) [] Received a spice Device without an address when processing VM 1325cebf-834e-4571-a3e1-82beb0021ffe devices, skipping device: {device=spice, specParams={fileTransferEnable=true, displayNetwork=yorku, copyPasteEnable=true, displayIp=130.63.94.35, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,ssmartcard,susbredir}, type=graphics, deviceId=3f758753-b44e-4641-a51c-e0554c983f33, tlsPort=5907} 2017-10-01 14:56:52,345-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmsStatisticsFetcher] (DefaultQuartzScheduler10) [4e27ca11] Fetched 10 VMs from VDS 'a0ccfd11-3402-4eba-a119-4987561b738e' 2017-10-01 14:56:52,345-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler10) [4e27ca11] VM 'db3e08fb-c111-442d-aeec-1507da30f486'(print) was unexpectedly detected as 'MigratingTo' on VDS 'a0ccfd11-3402-4eba-a119-4987561b738e'(virt2) (expected on '45f8b331-842e-48e7-9df8-56adddb93836') 2017-10-01 14:56:52,345-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler10) [4e27ca11] VM 'db3e08fb-c111-442d-aeec-1507da30f486' is migrating to VDS 'a0ccfd11-3402-4eba-a119-4987561b738e'(virt2) ignoring it in the refresh until migration is done 2017-10-01 14:56:54,903-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [e4aa1b2] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 14:56:55,499-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmsStatisticsFetcher] (DefaultQuartzScheduler8) [7de36b63] Fetched 4 VMs from VDS 'e1e71e9d-a0cd-47db-96c6-f724f6cea736' 2017-10-01 14:56:57,699-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmsStatisticsFetcher] (DefaultQuartzScheduler7) [5241cb01] Fetched 6 VMs from VDS '45f8b331-842e-48e7-9df8-56adddb93836' 2017-10-01 14:56:57,702-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler7) [5241cb01] VM '1325cebf-834e-4571-a3e1-82beb0021ffe' was reported as Down on VDS '45f8b331-842e-48e7-9df8-56adddb93836'(virt1) 2017-10-01 14:56:57,702-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (DefaultQuartzScheduler7) [5241cb01] START, DestroyVDSCommand(HostName = virt1, DestroyVmVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vmId='1325cebf-834e-4571-a3e1-82beb0021ffe', force='false', secondsToWait='0', gracefully='false', reason='', ignoreNoVm='true'}), log id: 5b640b36 2017-10-01 14:56:58,713-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (DefaultQuartzScheduler7) [5241cb01] FINISH, DestroyVDSCommand, log id: 5b640b36 2017-10-01 14:56:58,713-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler7) [5241cb01] VM '1325cebf-834e-4571-a3e1-82beb0021ffe'(dns1) was unexpectedly detected as 'Down' on VDS '45f8b331-842e-48e7-9df8-56adddb93836'(virt1) (expected on 'a0ccfd11-3402-4eba-a119-4987561b738e') 2017-10-01 14:57:07,367-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmsStatisticsFetcher] (DefaultQuartzScheduler10) [54d57f51] Fetched 11 VMs from VDS 'a0ccfd11-3402-4eba-a119-4987561b738e' 2017-10-01 14:57:07,368-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler10) [54d57f51] VM 'db3e08fb-c111-442d-aeec-1507da30f486'(print) was unexpectedly detected as 'MigratingTo' on VDS 'a0ccfd11-3402-4eba-a119-4987561b738e'(virt2) (expected on '45f8b331-842e-48e7-9df8-56adddb93836') 2017-10-01 14:57:07,368-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler10) [54d57f51] VM 'db3e08fb-c111-442d-aeec-1507da30f486' is migrating to VDS 'a0ccfd11-3402-4eba-a119-4987561b738e'(virt2) ignoring it in the refresh until migration is done 2017-10-01 14:57:07,369-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler10) [54d57f51] VM 'ddf37076-e795-4c87-a4fb-9efcaae76d72'(dc1) was unexpectedly detected as 'MigratingTo' on VDS 'a0ccfd11-3402-4eba-a119-4987561b738e'(virt2) (expected on '45f8b331-842e-48e7-9df8-56adddb93836') 2017-10-01 14:57:07,369-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler10) [54d57f51] VM 'ddf37076-e795-4c87-a4fb-9efcaae76d72' is migrating to VDS 'a0ccfd11-3402-4eba-a119-4987561b738e'(virt2) ignoring it in the refresh until migration is done 2017-10-01 14:57:13,729-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmsStatisticsFetcher] (DefaultQuartzScheduler8) [512559a3] Fetched 5 VMs from VDS '45f8b331-842e-48e7-9df8-56adddb93836' 2017-10-01 14:57:22,403-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler8) [6458673c] VM 'db3e08fb-c111-442d-aeec-1507da30f486'(print) was unexpectedly detected as 'MigratingTo' on VDS 'a0ccfd11-3402-4eba-a119-4987561b738e'(virt2) (expected on '45f8b331-842e-48e7-9df8-56adddb93836') 2017-10-01 14:57:22,403-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler8) [6458673c] VM 'db3e08fb-c111-442d-aeec-1507da30f486' is migrating to VDS 'a0ccfd11-3402-4eba-a119-4987561b738e'(virt2) ignoring it in the refresh until migration is done 2017-10-01 14:57:22,404-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler8) [6458673c] VM 'ddf37076-e795-4c87-a4fb-9efcaae76d72'(dc1) was unexpectedly detected as 'MigratingTo' on VDS 'a0ccfd11-3402-4eba-a119-4987561b738e'(virt2) (expected on '45f8b331-842e-48e7-9df8-56adddb93836') 2017-10-01 14:57:22,404-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler8) [6458673c] VM 'ddf37076-e795-4c87-a4fb-9efcaae76d72' is migrating to VDS 'a0ccfd11-3402-4eba-a119-4987561b738e'(virt2) ignoring it in the refresh until migration is done 2017-10-01 14:57:37,426-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler9) [64c3e963] VM 'db3e08fb-c111-442d-aeec-1507da30f486'(print) was unexpectedly detected as 'MigratingTo' on VDS 'a0ccfd11-3402-4eba-a119-4987561b738e'(virt2) (expected on '45f8b331-842e-48e7-9df8-56adddb93836') 2017-10-01 14:57:37,426-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler9) [64c3e963] VM 'db3e08fb-c111-442d-aeec-1507da30f486' is migrating to VDS 'a0ccfd11-3402-4eba-a119-4987561b738e'(virt2) ignoring it in the refresh until migration is done 2017-10-01 14:57:37,427-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler9) [64c3e963] VM 'ddf37076-e795-4c87-a4fb-9efcaae76d72'(dc1) was unexpectedly detected as 'MigratingTo' on VDS 'a0ccfd11-3402-4eba-a119-4987561b738e'(virt2) (expected on '45f8b331-842e-48e7-9df8-56adddb93836') 2017-10-01 14:57:37,427-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler9) [64c3e963] VM 'ddf37076-e795-4c87-a4fb-9efcaae76d72' is migrating to VDS 'a0ccfd11-3402-4eba-a119-4987561b738e'(virt2) ignoring it in the refresh until migration is done 2017-10-01 14:57:52,450-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler7) [482e4724] VM 'db3e08fb-c111-442d-aeec-1507da30f486'(print) was unexpectedly detected as 'MigratingTo' on VDS 'a0ccfd11-3402-4eba-a119-4987561b738e'(virt2) (expected on '45f8b331-842e-48e7-9df8-56adddb93836') 2017-10-01 14:57:52,450-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler7) [482e4724] VM 'db3e08fb-c111-442d-aeec-1507da30f486' is migrating to VDS 'a0ccfd11-3402-4eba-a119-4987561b738e'(virt2) ignoring it in the refresh until migration is done 2017-10-01 14:57:52,450-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler7) [482e4724] VM 'ddf37076-e795-4c87-a4fb-9efcaae76d72'(dc1) was unexpectedly detected as 'MigratingTo' on VDS 'a0ccfd11-3402-4eba-a119-4987561b738e'(virt2) (expected on '45f8b331-842e-48e7-9df8-56adddb93836') 2017-10-01 14:57:52,451-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler7) [482e4724] VM 'ddf37076-e795-4c87-a4fb-9efcaae76d72' is migrating to VDS 'a0ccfd11-3402-4eba-a119-4987561b738e'(virt2) ignoring it in the refresh until migration is done 2017-10-01 14:57:54,929-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [5d914301] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 14:57:57,911-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-11) [] VM 'db3e08fb-c111-442d-aeec-1507da30f486'(print) was unexpectedly detected as 'Up' on VDS 'a0ccfd11-3402-4eba-a119-4987561b738e'(virt2) (expected on '45f8b331-842e-48e7-9df8-56adddb93836') 2017-10-01 14:57:57,918-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ForkJoinPool-1-worker-11) [] EVENT_ID: VM_MIGRATION_DONE(63), Correlation ID: 34bc2b81, Job ID: 869f0fd4-4e10-4241-b2c0-d948f9643fbf, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Migration completed (VM: print, Source: virt1, Destination: virt2, Duration: 2 minutes 41 seconds, Total: 2 minutes 41 seconds, Actual downtime: 45ms) 2017-10-01 14:57:57,922-04 INFO [org.ovirt.engine.core.bll.MigrateVmCommand] (ForkJoinPool-1-worker-11) [] Lock freed to object 'EngineLock:{exclusiveLocks='[db3e08fb-c111-442d-aeec-1507da30f486=VM]', sharedLocks=''}' 2017-10-01 14:57:57,922-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVDSCommand] (ForkJoinPool-1-worker-11) [] START, FullListVDSCommand(HostName = virt2, FullListVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vmIds='[db3e08fb-c111-442d-aeec-1507da30f486]'}), log id: 407221b3 2017-10-01 14:57:59,770-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVDSCommand] (ForkJoinPool-1-worker-11) [] FINISH, FullListVDSCommand, return: [{acpiEnable=true, emulatedMachine=pc-i440fx-rhel7.3.0, afterMigrationStatus=, pid=37692, memGuaranteedSize=4096, transparentHugePages=true, timeOffset=-28799, cpuType=SandyBridge, smp=2, guestNumaNodes=[Ljava.lang.Object;@661ba362, smartcardEnable=false, custom={device_61d6b7fa-4074-4d11-9492-9d98f0092faf=VmDevice:{id='VmDeviceId:{deviceId='61d6b7fa-4074-4d11-9492-9d98f0092faf', vmId='db3e08fb-c111-442d-aeec-1507da30f486'}', device='ide', type='CONTROLLER', bootOrder='0', specParams='[]', address='{slot=0x01, bus=0x00, domain=0x0000, type=pci, function=0x1}', managed='false', plugged='true', readOnly='false', deviceAlias='ide', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_61d6b7fa-4074-4d11-9492-9d98f0092fafdevice_3fcd9941-2811-4e8d-a65d-c9191e0e73fbdevice_84b6cd65-a7af-4405-bc4f-144dd7b7b89cdevice_7f2f3fec-6aa5-4f44-9524-94f771eed4d3=VmDevice:{id='VmDeviceId:{deviceId='7f2f3fec-6aa5-4f44-9524-94f771eed4d3', vmId='db3e08fb-c111-442d-aeec-1507da30f486'}', device='spicevmc', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=3}', managed='false', plugged='true', readOnly='false', deviceAlias='channel2', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_61d6b7fa-4074-4d11-9492-9d98f0092fafdevice_3fcd9941-2811-4e8d-a65d-c9191e0e73fbdevice_84b6cd65-a7af-4405-bc4f-144dd7b7b89c=VmDevice:{id='VmDeviceId:{deviceId='84b6cd65-a7af-4405-bc4f-144dd7b7b89c', vmId='db3e08fb-c111-442d-aeec-1507da30f486'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=2}', managed='false', plugged='true', readOnly='false', deviceAlias='channel1', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_61d6b7fa-4074-4d11-9492-9d98f0092fafdevice_3fcd9941-2811-4e8d-a65d-c9191e0e73fb=VmDevice:{id='VmDeviceId:{deviceId='3fcd9941-2811-4e8d-a65d-c9191e0e73fb', vmId='db3e08fb-c111-442d-aeec-1507da30f486'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=1}', managed='false', plugged='true', readOnly='false', deviceAlias='channel0', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}}, vmType=kvm, memSize=4096, smpCoresPerSocket=1, vmName=print, nice=0, status=Up, maxMemSize=16384, bootMenuEnable=false, vmId=db3e08fb-c111-442d-aeec-1507da30f486, smpThreadsPerCore=1, guestDiskMapping={}, kvmEnable=true, displayNetwork=yorku, devices=[Ljava.lang.Object;@33ffe30b, display=qxl, maxVCpus=16, clientIp=, statusTime=4720961160, maxMemSlots=16}], log id: 407221b3 2017-10-01 14:57:59,771-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (ForkJoinPool-1-worker-11) [] Received a memballoon Device without an address when processing VM db3e08fb-c111-442d-aeec-1507da30f486 devices, skipping device: {device=memballoon, specParams={model=none}, type=balloon, target=4194304, alias=balloon0} 2017-10-01 14:57:59,771-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (ForkJoinPool-1-worker-11) [] Received a spice Device without an address when processing VM db3e08fb-c111-442d-aeec-1507da30f486 devices, skipping device: {device=spice, specParams={fileTransferEnable=true, displayNetwork=yorku, copyPasteEnable=true, displayIp=130.63.94.35, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,ssmartcard,susbredir}, type=graphics, deviceId=00000035-0035-0035-0035-0000000003a5, tlsPort=5908} 2017-10-01 14:57:59,782-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-11) [] VM 'ddf37076-e795-4c87-a4fb-9efcaae76d72'(dc1) was unexpectedly detected as 'Up' on VDS 'a0ccfd11-3402-4eba-a119-4987561b738e'(virt2) (expected on '45f8b331-842e-48e7-9df8-56adddb93836') 2017-10-01 14:57:59,787-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ForkJoinPool-1-worker-11) [] EVENT_ID: VM_MIGRATION_DONE(63), Correlation ID: 4232b55d, Job ID: 1e73de3d-36b7-4b23-9a56-735f7ff910d6, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Migration completed (VM: dc1, Source: virt1, Destination: virt2, Duration: 2 minutes 42 seconds, Total: 2 minutes 42 seconds, Actual downtime: 35ms) 2017-10-01 14:57:59,791-04 INFO [org.ovirt.engine.core.bll.MigrateVmCommand] (ForkJoinPool-1-worker-11) [] Lock freed to object 'EngineLock:{exclusiveLocks='[ddf37076-e795-4c87-a4fb-9efcaae76d72=VM]', sharedLocks=''}' 2017-10-01 14:57:59,791-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVDSCommand] (ForkJoinPool-1-worker-11) [] START, FullListVDSCommand(HostName = virt2, FullListVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vmIds='[ddf37076-e795-4c87-a4fb-9efcaae76d72]'}), log id: 36de10c4 2017-10-01 14:57:59,795-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler10) [5fbab742] VM 'db3e08fb-c111-442d-aeec-1507da30f486' was reported as Down on VDS '45f8b331-842e-48e7-9df8-56adddb93836'(virt1) 2017-10-01 14:57:59,795-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (DefaultQuartzScheduler10) [5fbab742] START, DestroyVDSCommand(HostName = virt1, DestroyVmVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vmId='db3e08fb-c111-442d-aeec-1507da30f486', force='false', secondsToWait='0', gracefully='false', reason='', ignoreNoVm='true'}), log id: 1f0f8dc4 2017-10-01 14:57:59,809-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (DefaultQuartzScheduler10) [5fbab742] FINISH, DestroyVDSCommand, log id: 1f0f8dc4 2017-10-01 14:57:59,810-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler10) [5fbab742] VM 'db3e08fb-c111-442d-aeec-1507da30f486'(print) was unexpectedly detected as 'Down' on VDS '45f8b331-842e-48e7-9df8-56adddb93836'(virt1) (expected on 'a0ccfd11-3402-4eba-a119-4987561b738e') 2017-10-01 14:57:59,810-04 WARN [org.ovirt.engine.core.vdsbroker.monitoring.VmsMonitoring] (DefaultQuartzScheduler10) [5fbab742] skipping VM 'ddf37076-e795-4c87-a4fb-9efcaae76d72' from this monitoring cycle - the VM data has changed since fetching the data 2017-10-01 14:57:59,810-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVDSCommand] (ForkJoinPool-1-worker-11) [] FINISH, FullListVDSCommand, return: [{acpiEnable=true, emulatedMachine=pc-i440fx-rhel7.3.0, afterMigrationStatus=, pid=37914, memGuaranteedSize=2730, transparentHugePages=true, timeOffset=1, cpuType=SandyBridge, smp=2, guestNumaNodes=[Ljava.lang.Object;@3b2e3de2, smartcardEnable=false, custom={device_c841b2bc-8eef-49ae-9cc4-6f969d141130device_d34252d5-5335-45b8-be7e-ae800e8cc457device_1f8acfa1-baf1-4a7e-80e3-ace473bb5a47device_3d3347cc-cc4f-4a78-9e00-620ab943e153=VmDevice:{id='VmDeviceId:{deviceId='3d3347cc-cc4f-4a78-9e00-620ab943e153', vmId='ddf37076-e795-4c87-a4fb-9efcaae76d72'}', device='spicevmc', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=3}', managed='false', plugged='true', readOnly='false', deviceAlias='channel2', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_c841b2bc-8eef-49ae-9cc4-6f969d141130=VmDevice:{id='VmDeviceId:{deviceId='c841b2bc-8eef-49ae-9cc4-6f969d141130', vmId='ddf37076-e795-4c87-a4fb-9efcaae76d72'}', device='ide', type='CONTROLLER', bootOrder='0', specParams='[]', address='{slot=0x01, bus=0x00, domain=0x0000, type=pci, function=0x1}', managed='false', plugged='true', readOnly='false', deviceAlias='ide', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_c841b2bc-8eef-49ae-9cc4-6f969d141130device_d34252d5-5335-45b8-be7e-ae800e8cc457device_1f8acfa1-baf1-4a7e-80e3-ace473bb5a47=VmDevice:{id='VmDeviceId:{deviceId='1f8acfa1-baf1-4a7e-80e3-ace473bb5a47', vmId='ddf37076-e795-4c87-a4fb-9efcaae76d72'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=2}', managed='false', plugged='true', readOnly='false', deviceAlias='channel1', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_c841b2bc-8eef-49ae-9cc4-6f969d141130device_d34252d5-5335-45b8-be7e-ae800e8cc457=VmDevice:{id='VmDeviceId:{deviceId='d34252d5-5335-45b8-be7e-ae800e8cc457', vmId='ddf37076-e795-4c87-a4fb-9efcaae76d72'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=1}', managed='false', plugged='true', readOnly='false', deviceAlias='channel0', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}}, vmType=kvm, memSize=4096, smpCoresPerSocket=1, vmName=dc1, nice=0, status=Up, maxMemSize=16384, bootMenuEnable=false, vmId=ddf37076-e795-4c87-a4fb-9efcaae76d72, smpThreadsPerCore=1, guestDiskMapping={}, kvmEnable=true, displayNetwork=yorku, devices=[Ljava.lang.Object;@25d28bd1, display=qxl, maxVCpus=16, clientIp=, statusTime=4720963030, maxMemSlots=16}], log id: 36de10c4 2017-10-01 14:57:59,811-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (ForkJoinPool-1-worker-11) [] Received a memballoon Device without an address when processing VM ddf37076-e795-4c87-a4fb-9efcaae76d72 devices, skipping device: {device=memballoon, specParams={model=none}, type=balloon, target=4194304, alias=balloon0} 2017-10-01 14:57:59,811-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (ForkJoinPool-1-worker-11) [] Received a spice Device without an address when processing VM ddf37076-e795-4c87-a4fb-9efcaae76d72 devices, skipping device: {device=spice, specParams={fileTransferEnable=true, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,ssmartcard,susbredir, copyPasteEnable=true, displayIp=130.63.94.35, displayNetwork=yorku}, type=graphics, deviceId=6fd350b3-e6f6-46ec-981d-d0250aebb024, tlsPort=5909} 2017-10-01 14:58:15,826-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmsStatisticsFetcher] (DefaultQuartzScheduler6) [5d31285e] Fetched 4 VMs from VDS '45f8b331-842e-48e7-9df8-56adddb93836' 2017-10-01 14:58:15,826-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler6) [5d31285e] VM 'ddf37076-e795-4c87-a4fb-9efcaae76d72' was reported as Down on VDS '45f8b331-842e-48e7-9df8-56adddb93836'(virt1) 2017-10-01 14:58:15,827-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (DefaultQuartzScheduler6) [5d31285e] START, DestroyVDSCommand(HostName = virt1, DestroyVmVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vmId='ddf37076-e795-4c87-a4fb-9efcaae76d72', force='false', secondsToWait='0', gracefully='false', reason='', ignoreNoVm='true'}), log id: 16ade71a 2017-10-01 14:58:16,254-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (DefaultQuartzScheduler6) [5d31285e] FINISH, DestroyVDSCommand, log id: 16ade71a 2017-10-01 14:58:16,254-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler6) [5d31285e] VM 'ddf37076-e795-4c87-a4fb-9efcaae76d72'(dc1) was unexpectedly detected as 'Down' on VDS '45f8b331-842e-48e7-9df8-56adddb93836'(virt1) (expected on 'a0ccfd11-3402-4eba-a119-4987561b738e') 2017-10-01 14:58:32,268-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmsStatisticsFetcher] (DefaultQuartzScheduler9) [62c7fe3e] Fetched 3 VMs from VDS '45f8b331-842e-48e7-9df8-56adddb93836' 2017-10-01 14:58:54,948-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [69130b12] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 14:59:54,967-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [41584dea] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 15:00:00,043-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [66e223cb] Autorecovering 2 hosts 2017-10-01 15:00:00,043-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [66e223cb] Autorecovering hosts id: e1e71e9d-a0cd-47db-96c6-f724f6cea736 , name : virt3 2017-10-01 15:00:00,051-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [5b0e88a4] Lock Acquired to object 'EngineLock:{exclusiveLocks='[e1e71e9d-a0cd-47db-96c6-f724f6cea736=VDS]', sharedLocks=''}' 2017-10-01 15:00:00,068-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [5b0e88a4] Running command: ActivateVdsCommand internal: true. Entities affected : ID: e1e71e9d-a0cd-47db-96c6-f724f6cea736 Type: VDSAction group MANIPULATE_HOST with role type ADMIN 2017-10-01 15:00:00,068-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [5b0e88a4] Before acquiring lock in order to prevent monitoring for host 'virt3' from data-center 'EECS-Primary' 2017-10-01 15:00:00,068-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [5b0e88a4] Lock acquired, from now a monitoring of host will be skipped for host 'virt3' from data-center 'EECS-Primary' 2017-10-01 15:00:00,084-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [5b0e88a4] START, SetVdsStatusVDSCommand(HostName = virt3, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2d2466be 2017-10-01 15:00:00,086-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [5b0e88a4] FINISH, SetVdsStatusVDSCommand, log id: 2d2466be 2017-10-01 15:00:00,091-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [5b0e88a4] Activate host finished. Lock released. Monitoring can run now for host 'virt3' from data-center 'EECS-Primary' 2017-10-01 15:00:00,100-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [5b0e88a4] EVENT_ID: VDS_ACTIVATE_ASYNC(9,502), Correlation ID: 5b0e88a4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host virt3 was autorecovered. 2017-10-01 15:00:00,100-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [5b0e88a4] Lock freed to object 'EngineLock:{exclusiveLocks='[e1e71e9d-a0cd-47db-96c6-f724f6cea736=VDS]', sharedLocks=''}' 2017-10-01 15:00:00,100-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [5b0e88a4] Autorecovering hosts id: 45f8b331-842e-48e7-9df8-56adddb93836 , name : virt1 2017-10-01 15:00:00,103-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [5fe85b43] Lock Acquired to object 'EngineLock:{exclusiveLocks='[45f8b331-842e-48e7-9df8-56adddb93836=VDS]', sharedLocks=''}' 2017-10-01 15:00:00,121-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [5fe85b43] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 45f8b331-842e-48e7-9df8-56adddb93836 Type: VDSAction group MANIPULATE_HOST with role type ADMIN 2017-10-01 15:00:00,121-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [5fe85b43] Before acquiring lock in order to prevent monitoring for host 'virt1' from data-center 'EECS-Primary' 2017-10-01 15:00:00,121-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [5fe85b43] Lock acquired, from now a monitoring of host will be skipped for host 'virt1' from data-center 'EECS-Primary' 2017-10-01 15:00:00,136-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [5fe85b43] START, SetVdsStatusVDSCommand(HostName = virt1, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3881489b 2017-10-01 15:00:00,138-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler2) [5fe85b43] FINISH, SetVdsStatusVDSCommand, log id: 3881489b 2017-10-01 15:00:00,142-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [5fe85b43] Activate host finished. Lock released. Monitoring can run now for host 'virt1' from data-center 'EECS-Primary' 2017-10-01 15:00:00,150-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [5fe85b43] EVENT_ID: VDS_ACTIVATE_ASYNC(9,502), Correlation ID: 5fe85b43, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host virt1 was autorecovered. 2017-10-01 15:00:00,150-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler2) [5fe85b43] Lock freed to object 'EngineLock:{exclusiveLocks='[45f8b331-842e-48e7-9df8-56adddb93836=VDS]', sharedLocks=''}' 2017-10-01 15:00:00,442-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [6b199276] START, GetHardwareInfoVDSCommand(HostName = virt1, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vds='Host[virt1,45f8b331-842e-48e7-9df8-56adddb93836]'}), log id: 8d92045 2017-10-01 15:00:00,506-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [6b199276] FINISH, GetHardwareInfoVDSCommand, log id: 8d92045 2017-10-01 15:00:00,609-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [233f8dd6] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 45f8b331-842e-48e7-9df8-56adddb93836 Type: VDS 2017-10-01 15:00:00,635-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [772076c9] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: aca61eac-ed8a-484b-bbee-3645148f6405 Type: StoragePool 2017-10-01 15:00:00,641-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [47c31afd] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: aca61eac-ed8a-484b-bbee-3645148f6405 Type: StoragePool 2017-10-01 15:00:00,655-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [47c31afd] START, ConnectStorageServerVDSCommand(HostName = virt1, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', storageType='NFS', connectionList='[StorageServerConnections:{id='5e557659-6817-40fa-9a55-7458125404a6', connection='virtfs-fs:/nfs/data', iqn='null', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='13cf1e67-9961-4a25-a41f-cc76b4ab793d', connection='virtfs-fs:/nfs/iso', iqn='null', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='417fdd51-dbcd-41d9-9a05-ef4d9f485d13', connection='virtfs-fs:/nfs/export', iqn='null', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 29aa12fa 2017-10-01 15:00:01,143-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [47c31afd] FINISH, ConnectStorageServerVDSCommand, return: {5e557659-6817-40fa-9a55-7458125404a6=0, 417fdd51-dbcd-41d9-9a05-ef4d9f485d13=0, 13cf1e67-9961-4a25-a41f-cc76b4ab793d=0}, log id: 29aa12fa 2017-10-01 15:00:01,148-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [47c31afd] Host 'virt1' storage connection was succeeded 2017-10-01 15:00:01,159-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] START, ConnectStoragePoolVDSCommand(HostName = virt1, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vdsId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 39928ade 2017-10-01 15:00:01,159-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 15:00:01,312-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [7d564f15] START, GetHardwareInfoVDSCommand(HostName = virt3, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vds='Host[virt3,e1e71e9d-a0cd-47db-96c6-f724f6cea736]'}), log id: 68be48d 2017-10-01 15:00:01,328-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [7d564f15] FINISH, GetHardwareInfoVDSCommand, log id: 68be48d 2017-10-01 15:00:01,419-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [63603175] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: e1e71e9d-a0cd-47db-96c6-f724f6cea736 Type: VDS 2017-10-01 15:00:01,445-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [723abd4c] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: aca61eac-ed8a-484b-bbee-3645148f6405 Type: StoragePool 2017-10-01 15:00:01,451-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [7555b7b8] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: aca61eac-ed8a-484b-bbee-3645148f6405 Type: StoragePool 2017-10-01 15:00:01,463-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [7555b7b8] START, ConnectStorageServerVDSCommand(HostName = virt3, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', storageType='NFS', connectionList='[StorageServerConnections:{id='5e557659-6817-40fa-9a55-7458125404a6', connection='virtfs-fs:/nfs/data', iqn='null', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='13cf1e67-9961-4a25-a41f-cc76b4ab793d', connection='virtfs-fs:/nfs/iso', iqn='null', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='417fdd51-dbcd-41d9-9a05-ef4d9f485d13', connection='virtfs-fs:/nfs/export', iqn='null', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 2c665669 2017-10-01 15:00:01,466-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-28) [] FINISH, ConnectStoragePoolVDSCommand, log id: 39928ade 2017-10-01 15:00:01,663-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [47c31afd] EVENT_ID: FENCE_OPERATION_USING_AGENT_AND_PROXY_STARTED(9,020), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Executing power management status on Host virt1 using Proxy Host virt4 and Fence Agent ipmilan:virt1-idrac. 2017-10-01 15:00:01,663-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler1) [47c31afd] START, FenceVdsVDSCommand(HostName = virt4, FenceVdsVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', targetVdsId='45f8b331-842e-48e7-9df8-56adddb93836', action='STATUS', agent='FenceAgent:{id='b92fdde5-0407-4d57-8d00-108a0cea8300', hostId='45f8b331-842e-48e7-9df8-56adddb93836', order='1', type='ipmilan', ip='virt1-idrac', port='null', user='root', password='***', encryptOptions='false', options='privlvl=OPERATOR, delay=10, lanplus=1'}', policy='null'}), log id: 91800d5 2017-10-01 15:00:01,671-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [7555b7b8] FINISH, ConnectStorageServerVDSCommand, return: {5e557659-6817-40fa-9a55-7458125404a6=0, 417fdd51-dbcd-41d9-9a05-ef4d9f485d13=0, 13cf1e67-9961-4a25-a41f-cc76b4ab793d=0}, log id: 2c665669 2017-10-01 15:00:01,675-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [7555b7b8] Host 'virt3' storage connection was succeeded 2017-10-01 15:00:01,686-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] START, ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 607fe07 2017-10-01 15:00:01,686-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 15:00:01,867-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler1) [47c31afd] FINISH, FenceVdsVDSCommand, return: FenceOperationResult:{status='SUCCESS', powerStatus='ON', message=''}, log id: 91800d5 2017-10-01 15:00:01,882-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler1) [47c31afd] START, SetMOMPolicyParametersVDSCommand(HostName = virt1, MomPolicyVDSParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836'}), log id: 105ef817 2017-10-01 15:00:01,888-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] FINISH, ConnectStoragePoolVDSCommand, log id: 607fe07 2017-10-01 15:00:02,070-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [7555b7b8] EVENT_ID: FENCE_OPERATION_USING_AGENT_AND_PROXY_STARTED(9,020), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Executing power management status on Host virt3 using Proxy Host virt4 and Fence Agent ipmilan:virt3-idrac. 2017-10-01 15:00:02,070-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler8) [7555b7b8] START, FenceVdsVDSCommand(HostName = virt4, FenceVdsVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', targetVdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', action='STATUS', agent='FenceAgent:{id='8c639beb-546a-4e77-9c21-1ffb54f1daae', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', order='1', type='ipmilan', ip='virt3-idrac', port='null', user='root', password='***', encryptOptions='false', options='privlvl=OPERATOR, delay=10, lanplus=1'}', policy='null'}), log id: 41e71d50 2017-10-01 15:00:02,073-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler1) [47c31afd] FINISH, SetMOMPolicyParametersVDSCommand, log id: 105ef817 2017-10-01 15:00:02,092-04 INFO [org.ovirt.engine.core.bll.hostdev.RefreshHostDevicesCommand] (DefaultQuartzScheduler1) [5ba91015] Running command: RefreshHostDevicesCommand internal: true. Entities affected : ID: 45f8b331-842e-48e7-9df8-56adddb93836 Type: VDSAction group MANIPULATE_HOST with role type ADMIN 2017-10-01 15:00:02,310-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler8) [7555b7b8] FINISH, FenceVdsVDSCommand, return: FenceOperationResult:{status='SUCCESS', powerStatus='ON', message=''}, log id: 41e71d50 2017-10-01 15:00:02,325-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler8) [7555b7b8] START, SetMOMPolicyParametersVDSCommand(HostName = virt3, MomPolicyVDSParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736'}), log id: 68597276 2017-10-01 15:00:02,343-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [5ba91015] EVENT_ID: VDS_DETECTED(13), Correlation ID: 772076c9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Status of host virt1 was set to Up. 2017-10-01 15:00:02,345-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [5ba91015] EVENT_ID: VDS_FENCE_STATUS(496), Correlation ID: 772076c9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host virt1 power management was verified successfully. 2017-10-01 15:00:02,376-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [30b11828] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 45f8b331-842e-48e7-9df8-56adddb93836 Type: VDS 2017-10-01 15:00:03,392-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler8) [7555b7b8] FINISH, SetMOMPolicyParametersVDSCommand, log id: 68597276 2017-10-01 15:00:03,411-04 INFO [org.ovirt.engine.core.bll.hostdev.RefreshHostDevicesCommand] (DefaultQuartzScheduler8) [be9d7ed] Running command: RefreshHostDevicesCommand internal: true. Entities affected : ID: e1e71e9d-a0cd-47db-96c6-f724f6cea736 Type: VDSAction group MANIPULATE_HOST with role type ADMIN 2017-10-01 15:00:03,573-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [be9d7ed] EVENT_ID: VDS_DETECTED(13), Correlation ID: 723abd4c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Status of host virt3 was set to Up. 2017-10-01 15:00:03,575-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [be9d7ed] EVENT_ID: VDS_FENCE_STATUS(496), Correlation ID: 723abd4c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host virt3 power management was verified successfully. 2017-10-01 15:00:03,602-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [316b4e81] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: e1e71e9d-a0cd-47db-96c6-f724f6cea736 Type: VDS 2017-10-01 15:00:54,986-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [5ef48055] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 15:01:39,925-04 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-8) [] domain 'f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2:export' in problem 'PROBLEMATIC'. vds: 'virt3' 2017-10-01 15:01:39,926-04 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-8) [] domain 'c8653dba-63ec-4447-872c-0ffcc6af9c0f:data' in problem 'PROBLEMATIC'. vds: 'virt3' 2017-10-01 15:01:39,927-04 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-8) [] domain '63d680c7-4c94-4b29-a6cb-f288a128b9bc:iso' in problem 'PROBLEMATIC'. vds: 'virt3' 2017-10-01 15:01:55,007-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [258ec99a] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 15:02:11,191-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler5) [1795a43f] VM '1734fed5-fc97-4714-bb63-55e8f4eb6d34'(wiki) moved from 'Up' --> 'NotResponding' 2017-10-01 15:02:11,265-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [1795a43f] EVENT_ID: VM_NOT_RESPONDING(126), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM wiki is not responding. 2017-10-01 15:02:11,265-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler5) [1795a43f] VM 'cc1dbe93-bb80-451e-8b3b-c9cc5e90432f'(license) moved from 'Up' --> 'NotResponding' 2017-10-01 15:02:11,287-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [1795a43f] EVENT_ID: VM_NOT_RESPONDING(126), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM license is not responding. 2017-10-01 15:02:11,287-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler5) [1795a43f] VM '64734fca-4d6c-443c-82d1-abf7cf106601'(webdav) moved from 'Up' --> 'NotResponding' 2017-10-01 15:02:11,308-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [1795a43f] EVENT_ID: VM_NOT_RESPONDING(126), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM webdav is not responding. 2017-10-01 15:02:55,029-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [26f2a6bf] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 15:03:55,049-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [feac8e8] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 15:04:55,070-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [4e463c55] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 15:05:55,089-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [538e770] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 15:06:39,927-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-19) [] starting processDomainRecovery for domain 'c8653dba-63ec-4447-872c-0ffcc6af9c0f:data'. 2017-10-01 15:06:39,983-04 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-19) [] vds 'virt3' reported domain 'c8653dba-63ec-4447-872c-0ffcc6af9c0f:data' as in problem, attempting to move the vds to status NonOperational 2017-10-01 15:06:39,984-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-19) [] Domain 'f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2:export' has recovered from problem. No active host in the DC is reporting it as poblematic, so clearing the domain recovery timer. 2017-10-01 15:06:39,985-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-19) [] Domain '63d680c7-4c94-4b29-a6cb-f288a128b9bc:iso' has recovered from problem. No active host in the DC is reporting it as poblematic, so clearing the domain recovery timer. 2017-10-01 15:06:39,985-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-19) [] Removing vds '[e1e71e9d-a0cd-47db-96c6-f724f6cea736]' from the domain in maintenance cache 2017-10-01 15:06:39,985-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-19) [] Removing host(s) '[e1e71e9d-a0cd-47db-96c6-f724f6cea736]' from hosts unseen domain report cache 2017-10-01 15:06:40,016-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (org.ovirt.thread.pool-6-thread-47) [50579690] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: e1e71e9d-a0cd-47db-96c6-f724f6cea736 Type: VDS 2017-10-01 15:06:40,031-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-6-thread-47) [50579690] START, SetVdsStatusVDSCommand(HostName = virt3, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7bb75401 2017-10-01 15:06:40,033-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-6-thread-47) [50579690] FINISH, SetVdsStatusVDSCommand, log id: 7bb75401 2017-10-01 15:06:40,090-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-47) [50579690] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Correlation ID: 50579690, Job ID: 33233790-2bce-4d44-8520-1837da71f44e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host virt3 cannot access the Storage Domain(s) data attached to the Data Center EECS-Primary. Setting Host state to Non-Operational. 2017-10-01 15:06:40,140-04 INFO [org.ovirt.engine.core.bll.MigrateVmCommand] (org.ovirt.thread.pool-6-thread-12) [46ebc516] Lock Acquired to object 'EngineLock:{exclusiveLocks='[c64d80f9-a3bd-44f3-89c4-f1b9b509cdde=VM]', sharedLocks=''}' 2017-10-01 15:06:40,156-04 WARN [org.ovirt.engine.core.bll.MigrateVmCommand] (org.ovirt.thread.pool-6-thread-12) [46ebc516] Validation of action 'MigrateVm' failed for user SYSTEM. Reasons: VAR__ACTION__MIGRATE,VAR__TYPE__VM,ACTION_TYPE_FAILED_VM_STATUS_ILLEGAL,VAR__VM_STATUS__NOT_RESPONDING 2017-10-01 15:06:40,157-04 INFO [org.ovirt.engine.core.bll.MigrateVmCommand] (org.ovirt.thread.pool-6-thread-12) [46ebc516] Lock freed to object 'EngineLock:{exclusiveLocks='[c64d80f9-a3bd-44f3-89c4-f1b9b509cdde=VM]', sharedLocks=''}' 2017-10-01 15:06:40,162-04 ERROR [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (org.ovirt.thread.pool-6-thread-12) [46ebc516] Failed to migrate VM 'red' 2017-10-01 15:06:40,178-04 INFO [org.ovirt.engine.core.bll.MigrateVmCommand] (org.ovirt.thread.pool-6-thread-12) [37e60211] Lock Acquired to object 'EngineLock:{exclusiveLocks='[1734fed5-fc97-4714-bb63-55e8f4eb6d34=VM]', sharedLocks=''}' 2017-10-01 15:06:40,194-04 WARN [org.ovirt.engine.core.bll.MigrateVmCommand] (org.ovirt.thread.pool-6-thread-12) [37e60211] Validation of action 'MigrateVm' failed for user SYSTEM. Reasons: VAR__ACTION__MIGRATE,VAR__TYPE__VM,ACTION_TYPE_FAILED_VM_STATUS_ILLEGAL,VAR__VM_STATUS__NOT_RESPONDING 2017-10-01 15:06:40,194-04 INFO [org.ovirt.engine.core.bll.MigrateVmCommand] (org.ovirt.thread.pool-6-thread-12) [37e60211] Lock freed to object 'EngineLock:{exclusiveLocks='[1734fed5-fc97-4714-bb63-55e8f4eb6d34=VM]', sharedLocks=''}' 2017-10-01 15:06:40,198-04 ERROR [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (org.ovirt.thread.pool-6-thread-12) [37e60211] Failed to migrate VM 'wiki' 2017-10-01 15:06:40,208-04 INFO [org.ovirt.engine.core.bll.MigrateVmCommand] (org.ovirt.thread.pool-6-thread-12) [e38dbe] Lock Acquired to object 'EngineLock:{exclusiveLocks='[cc1dbe93-bb80-451e-8b3b-c9cc5e90432f=VM]', sharedLocks=''}' 2017-10-01 15:06:40,223-04 WARN [org.ovirt.engine.core.bll.MigrateVmCommand] (org.ovirt.thread.pool-6-thread-12) [e38dbe] Validation of action 'MigrateVm' failed for user SYSTEM. Reasons: VAR__ACTION__MIGRATE,VAR__TYPE__VM,ACTION_TYPE_FAILED_VM_STATUS_ILLEGAL,VAR__VM_STATUS__NOT_RESPONDING 2017-10-01 15:06:40,224-04 INFO [org.ovirt.engine.core.bll.MigrateVmCommand] (org.ovirt.thread.pool-6-thread-12) [e38dbe] Lock freed to object 'EngineLock:{exclusiveLocks='[cc1dbe93-bb80-451e-8b3b-c9cc5e90432f=VM]', sharedLocks=''}' 2017-10-01 15:06:40,228-04 ERROR [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (org.ovirt.thread.pool-6-thread-12) [e38dbe] Failed to migrate VM 'license' 2017-10-01 15:06:40,245-04 INFO [org.ovirt.engine.core.bll.MigrateVmCommand] (org.ovirt.thread.pool-6-thread-12) [42db3fb8] Lock Acquired to object 'EngineLock:{exclusiveLocks='[64734fca-4d6c-443c-82d1-abf7cf106601=VM]', sharedLocks=''}' 2017-10-01 15:06:40,261-04 WARN [org.ovirt.engine.core.bll.MigrateVmCommand] (org.ovirt.thread.pool-6-thread-12) [42db3fb8] Validation of action 'MigrateVm' failed for user SYSTEM. Reasons: VAR__ACTION__MIGRATE,VAR__TYPE__VM,ACTION_TYPE_FAILED_VM_STATUS_ILLEGAL,VAR__VM_STATUS__NOT_RESPONDING 2017-10-01 15:06:40,261-04 INFO [org.ovirt.engine.core.bll.MigrateVmCommand] (org.ovirt.thread.pool-6-thread-12) [42db3fb8] Lock freed to object 'EngineLock:{exclusiveLocks='[64734fca-4d6c-443c-82d1-abf7cf106601=VM]', sharedLocks=''}' 2017-10-01 15:06:40,265-04 ERROR [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (org.ovirt.thread.pool-6-thread-12) [42db3fb8] Failed to migrate VM 'webdav' 2017-10-01 15:06:55,108-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [47fa308a] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 15:07:23,495-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler9) [10b01bbe] VM '64734fca-4d6c-443c-82d1-abf7cf106601'(webdav) moved from 'NotResponding' --> 'Up' 2017-10-01 15:07:38,527-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler9) [38f62baf] VM '1734fed5-fc97-4714-bb63-55e8f4eb6d34'(wiki) moved from 'NotResponding' --> 'Up' 2017-10-01 15:07:53,553-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler4) [8e55bb5] VM 'cc1dbe93-bb80-451e-8b3b-c9cc5e90432f'(license) moved from 'NotResponding' --> 'Up' 2017-10-01 15:07:55,132-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [296bd98c] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 15:08:55,152-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [5fc00b92] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 15:09:55,175-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [5f7c09b7] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 15:10:00,025-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [71220e1f] Autorecovering 1 hosts 2017-10-01 15:10:00,026-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler5) [71220e1f] Autorecovering hosts id: e1e71e9d-a0cd-47db-96c6-f724f6cea736 , name : virt3 2017-10-01 15:10:00,031-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [5d7282e1] Lock Acquired to object 'EngineLock:{exclusiveLocks='[e1e71e9d-a0cd-47db-96c6-f724f6cea736=VDS]', sharedLocks=''}' 2017-10-01 15:10:00,050-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [5d7282e1] Running command: ActivateVdsCommand internal: true. Entities affected : ID: e1e71e9d-a0cd-47db-96c6-f724f6cea736 Type: VDSAction group MANIPULATE_HOST with role type ADMIN 2017-10-01 15:10:00,050-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [5d7282e1] Before acquiring lock in order to prevent monitoring for host 'virt3' from data-center 'EECS-Primary' 2017-10-01 15:10:00,050-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [5d7282e1] Lock acquired, from now a monitoring of host will be skipped for host 'virt3' from data-center 'EECS-Primary' 2017-10-01 15:10:00,066-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [5d7282e1] START, SetVdsStatusVDSCommand(HostName = virt3, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3ed23546 2017-10-01 15:10:00,068-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler5) [5d7282e1] FINISH, SetVdsStatusVDSCommand, log id: 3ed23546 2017-10-01 15:10:00,074-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [5d7282e1] Activate host finished. Lock released. Monitoring can run now for host 'virt3' from data-center 'EECS-Primary' 2017-10-01 15:10:00,077-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler5) [5d7282e1] Lock freed to object 'EngineLock:{exclusiveLocks='[e1e71e9d-a0cd-47db-96c6-f724f6cea736=VDS]', sharedLocks=''}' 2017-10-01 15:10:03,909-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [20638bc0] START, GetHardwareInfoVDSCommand(HostName = virt3, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vds='Host[virt3,e1e71e9d-a0cd-47db-96c6-f724f6cea736]'}), log id: 3622ee7d 2017-10-01 15:10:04,918-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [20638bc0] FINISH, GetHardwareInfoVDSCommand, log id: 3622ee7d 2017-10-01 15:10:05,014-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [56249b53] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: e1e71e9d-a0cd-47db-96c6-f724f6cea736 Type: VDS 2017-10-01 15:10:05,040-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [7435243f] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: aca61eac-ed8a-484b-bbee-3645148f6405 Type: StoragePool 2017-10-01 15:10:05,047-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [48140b27] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: aca61eac-ed8a-484b-bbee-3645148f6405 Type: StoragePool 2017-10-01 15:10:05,060-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [48140b27] START, ConnectStorageServerVDSCommand(HostName = virt3, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', storageType='NFS', connectionList='[StorageServerConnections:{id='5e557659-6817-40fa-9a55-7458125404a6', connection='virtfs-fs:/nfs/data', iqn='null', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='13cf1e67-9961-4a25-a41f-cc76b4ab793d', connection='virtfs-fs:/nfs/iso', iqn='null', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='417fdd51-dbcd-41d9-9a05-ef4d9f485d13', connection='virtfs-fs:/nfs/export', iqn='null', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 58f8cef1 2017-10-01 15:10:05,674-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [48140b27] FINISH, ConnectStorageServerVDSCommand, return: {5e557659-6817-40fa-9a55-7458125404a6=0, 417fdd51-dbcd-41d9-9a05-ef4d9f485d13=0, 13cf1e67-9961-4a25-a41f-cc76b4ab793d=0}, log id: 58f8cef1 2017-10-01 15:10:05,678-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [48140b27] Host 'virt3' storage connection was succeeded 2017-10-01 15:10:05,689-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] START, ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 6cedef28 2017-10-01 15:10:05,689-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 15:10:06,861-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-6) [] FINISH, ConnectStoragePoolVDSCommand, log id: 6cedef28 2017-10-01 15:10:08,056-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [48140b27] EVENT_ID: FENCE_OPERATION_USING_AGENT_AND_PROXY_STARTED(9,020), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Executing power management status on Host virt3 using Proxy Host virt1 and Fence Agent ipmilan:virt3-idrac. 2017-10-01 15:10:08,056-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler3) [48140b27] START, FenceVdsVDSCommand(HostName = virt1, FenceVdsVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', targetVdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', action='STATUS', agent='FenceAgent:{id='8c639beb-546a-4e77-9c21-1ffb54f1daae', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', order='1', type='ipmilan', ip='virt3-idrac', port='null', user='root', password='***', encryptOptions='false', options='privlvl=OPERATOR, delay=10, lanplus=1'}', policy='null'}), log id: 50d0a779 2017-10-01 15:10:08,782-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler3) [48140b27] FINISH, FenceVdsVDSCommand, return: FenceOperationResult:{status='SUCCESS', powerStatus='ON', message=''}, log id: 50d0a779 2017-10-01 15:10:08,796-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler3) [48140b27] START, SetMOMPolicyParametersVDSCommand(HostName = virt3, MomPolicyVDSParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736'}), log id: 5320b976 2017-10-01 15:10:10,714-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler3) [48140b27] FINISH, SetMOMPolicyParametersVDSCommand, log id: 5320b976 2017-10-01 15:10:10,732-04 INFO [org.ovirt.engine.core.bll.hostdev.RefreshHostDevicesCommand] (DefaultQuartzScheduler3) [54a30d82] Running command: RefreshHostDevicesCommand internal: true. Entities affected : ID: e1e71e9d-a0cd-47db-96c6-f724f6cea736 Type: VDSAction group MANIPULATE_HOST with role type ADMIN 2017-10-01 15:10:10,797-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [54a30d82] EVENT_ID: VDS_DETECTED(13), Correlation ID: 7435243f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Status of host virt3 was set to Up. 2017-10-01 15:10:10,800-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [54a30d82] EVENT_ID: VDS_FENCE_STATUS(496), Correlation ID: 7435243f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host virt3 power management was verified successfully. 2017-10-01 15:10:10,830-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [505c48b5] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: e1e71e9d-a0cd-47db-96c6-f724f6cea736 Type: VDS 2017-10-01 15:10:55,197-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [703fc386] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 15:11:55,217-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [3f5645a0] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 15:12:55,242-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [3312a9e1] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 15:13:55,263-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [57f8339c] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 15:14:55,280-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [54d76a11] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 15:15:55,299-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [37f07aeb] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 15:16:55,318-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [12dd9205] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 15:17:55,342-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [1b0f7ba8] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 15:18:55,361-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler7) [3b27b32c] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 15:19:55,380-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [1831ad48] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 15:20:24,685-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.OvfDataUpdater] (DefaultQuartzScheduler1) [6b00677a] Attempting to update VMs/Templates Ovf. 2017-10-01 15:20:24,688-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler1) [3a36e15f] Before acquiring and wait lock 'EngineLock:{exclusiveLocks='[aca61eac-ed8a-484b-bbee-3645148f6405=OVF_UPDATE]', sharedLocks=''}' 2017-10-01 15:20:24,688-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler1) [3a36e15f] Lock-wait acquired to object 'EngineLock:{exclusiveLocks='[aca61eac-ed8a-484b-bbee-3645148f6405=OVF_UPDATE]', sharedLocks=''}' 2017-10-01 15:20:24,689-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler1) [3a36e15f] Running command: ProcessOvfUpdateForStoragePoolCommand internal: true. Entities affected : ID: aca61eac-ed8a-484b-bbee-3645148f6405 Type: StoragePool 2017-10-01 15:20:24,752-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler1) [3a36e15f] Attempting to update VM OVFs in Data Center 'EECS-Primary' 2017-10-01 15:20:24,933-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler1) [3a36e15f] Successfully updated VM OVFs in Data Center 'EECS-Primary' 2017-10-01 15:20:24,934-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler1) [3a36e15f] Attempting to update template OVFs in Data Center 'EECS-Primary' 2017-10-01 15:20:24,935-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler1) [3a36e15f] Successfully updated templates OVFs in Data Center 'EECS-Primary' 2017-10-01 15:20:24,935-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler1) [3a36e15f] Attempting to remove unneeded template/vm OVFs in Data Center 'EECS-Primary' 2017-10-01 15:20:24,937-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler1) [3a36e15f] Successfully removed unneeded template/vm OVFs in Data Center 'EECS-Primary' 2017-10-01 15:20:24,938-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler1) [3a36e15f] Lock freed to object 'EngineLock:{exclusiveLocks='[aca61eac-ed8a-484b-bbee-3645148f6405=OVF_UPDATE]', sharedLocks=''}' 2017-10-01 15:20:24,963-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStorageDomainCommand] (DefaultQuartzScheduler1) [35bcb02d] Lock Acquired to object 'EngineLock:{exclusiveLocks='[c8653dba-63ec-4447-872c-0ffcc6af9c0f=STORAGE]', sharedLocks='[aca61eac-ed8a-484b-bbee-3645148f6405=OVF_UPDATE]'}' 2017-10-01 15:20:24,991-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStorageDomainCommand] (DefaultQuartzScheduler1) [35bcb02d] Running command: ProcessOvfUpdateForStorageDomainCommand internal: true. Entities affected : ID: c8653dba-63ec-4447-872c-0ffcc6af9c0f Type: StorageAction group MANIPULATE_STORAGE_DOMAIN with role type ADMIN 2017-10-01 15:20:25,007-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler1) [35bcb02d] START, SetVolumeDescriptionVDSCommand( SetVolumeDescriptionVDSCommandParameters:{runAsync='true', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', ignoreFailoverLimit='false', storageDomainId='c8653dba-63ec-4447-872c-0ffcc6af9c0f', imageGroupId='bbbcf706-d019-4cc5-8096-ddf6d9436553', imageId='6c73cd57-01c6-4541-8157-495e5182f916'}), log id: 67637448 2017-10-01 15:20:25,007-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler1) [35bcb02d] -- executeIrsBrokerCommand: calling 'setVolumeDescription', parameters: 2017-10-01 15:20:25,007-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler1) [35bcb02d] ++ spUUID=aca61eac-ed8a-484b-bbee-3645148f6405 2017-10-01 15:20:25,007-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler1) [35bcb02d] ++ sdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f 2017-10-01 15:20:25,007-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler1) [35bcb02d] ++ imageGroupGUID=bbbcf706-d019-4cc5-8096-ddf6d9436553 2017-10-01 15:20:25,007-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler1) [35bcb02d] ++ volUUID=6c73cd57-01c6-4541-8157-495e5182f916 2017-10-01 15:20:25,007-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler1) [35bcb02d] ++ description={"Updated":false,"Last Updated":"Wed Sep 27 10:20:12 EDT 2017","Storage Domains":[{"uuid":"c8653dba-63ec-4447-872c-0ffcc6af9c0f"}],"Disk Description":"OVF_STORE"} 2017-10-01 15:20:26,039-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler1) [35bcb02d] FINISH, SetVolumeDescriptionVDSCommand, log id: 67637448 2017-10-01 15:20:26,060-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.UploadStreamCommand] (DefaultQuartzScheduler1) [35bcb02d] Lock Acquired to object 'EngineLock:{exclusiveLocks='', sharedLocks='[5ae29ed2-9c0f-4754-8df3-b43d14726846=VDS_EXECUTION]'}' 2017-10-01 15:20:26,075-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.UploadStreamCommand] (DefaultQuartzScheduler1) [35bcb02d] Running command: UploadStreamCommand internal: true. Entities affected : ID: c8653dba-63ec-4447-872c-0ffcc6af9c0f Type: Storage 2017-10-01 15:20:26,075-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (DefaultQuartzScheduler1) [35bcb02d] START, UploadStreamVDSCommand(HostName = virt4, UploadStreamVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846'}), log id: 191a9bbb 2017-10-01 15:20:26,075-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (DefaultQuartzScheduler1) [35bcb02d] -- executeVdsBrokerCommand, parameters: 2017-10-01 15:20:26,075-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (DefaultQuartzScheduler1) [35bcb02d] ++ spUUID=aca61eac-ed8a-484b-bbee-3645148f6405 2017-10-01 15:20:26,075-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (DefaultQuartzScheduler1) [35bcb02d] ++ sdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f 2017-10-01 15:20:26,075-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (DefaultQuartzScheduler1) [35bcb02d] ++ imageGUID=bbbcf706-d019-4cc5-8096-ddf6d9436553 2017-10-01 15:20:26,075-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (DefaultQuartzScheduler1) [35bcb02d] ++ volUUID=6c73cd57-01c6-4541-8157-495e5182f916 2017-10-01 15:20:26,075-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (DefaultQuartzScheduler1) [35bcb02d] ++ size=481280 2017-10-01 15:20:27,222-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (DefaultQuartzScheduler1) [35bcb02d] FINISH, UploadStreamVDSCommand, log id: 191a9bbb 2017-10-01 15:20:27,236-04 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler1) [35bcb02d] CommandAsyncTask::Adding CommandMultiAsyncTasks object for command '756c18c8-aded-4699-b45d-42242da0118c' 2017-10-01 15:20:27,236-04 INFO [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (DefaultQuartzScheduler1) [35bcb02d] CommandMultiAsyncTasks::attachTask: Attaching task '0fa9c866-7406-4972-9cab-eea5eaeed8ca' to command '756c18c8-aded-4699-b45d-42242da0118c'. 2017-10-01 15:20:27,248-04 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler1) [35bcb02d] Adding task '0fa9c866-7406-4972-9cab-eea5eaeed8ca' (Parent Command 'UploadStream', Parameters Type 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters'), polling hasn't started yet.. 2017-10-01 15:20:27,253-04 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler1) [35bcb02d] BaseAsyncTask::startPollingTask: Starting to poll task '0fa9c866-7406-4972-9cab-eea5eaeed8ca'. 2017-10-01 15:20:27,257-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.UploadStreamCommand] (DefaultQuartzScheduler1) [35bcb02d] Lock freed to object 'EngineLock:{exclusiveLocks='', sharedLocks='[5ae29ed2-9c0f-4754-8df3-b43d14726846=VDS_EXECUTION]'}' 2017-10-01 15:20:27,259-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler1) [35bcb02d] START, SetVolumeDescriptionVDSCommand( SetVolumeDescriptionVDSCommandParameters:{runAsync='true', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', ignoreFailoverLimit='false', storageDomainId='c8653dba-63ec-4447-872c-0ffcc6af9c0f', imageGroupId='bbbcf706-d019-4cc5-8096-ddf6d9436553', imageId='6c73cd57-01c6-4541-8157-495e5182f916'}), log id: 2c84ef9a 2017-10-01 15:20:27,259-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler1) [35bcb02d] -- executeIrsBrokerCommand: calling 'setVolumeDescription', parameters: 2017-10-01 15:20:27,259-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler1) [35bcb02d] ++ spUUID=aca61eac-ed8a-484b-bbee-3645148f6405 2017-10-01 15:20:27,259-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler1) [35bcb02d] ++ sdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f 2017-10-01 15:20:27,259-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler1) [35bcb02d] ++ imageGroupGUID=bbbcf706-d019-4cc5-8096-ddf6d9436553 2017-10-01 15:20:27,259-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler1) [35bcb02d] ++ volUUID=6c73cd57-01c6-4541-8157-495e5182f916 2017-10-01 15:20:27,259-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler1) [35bcb02d] ++ description={"Updated":true,"Size":481280,"Last Updated":"Sun Oct 01 15:20:24 EDT 2017","Storage Domains":[{"uuid":"c8653dba-63ec-4447-872c-0ffcc6af9c0f"}],"Disk Description":"OVF_STORE"} 2017-10-01 15:20:28,261-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler1) [35bcb02d] FINISH, SetVolumeDescriptionVDSCommand, log id: 2c84ef9a 2017-10-01 15:20:28,263-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler1) [35bcb02d] START, SetVolumeDescriptionVDSCommand( SetVolumeDescriptionVDSCommandParameters:{runAsync='true', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', ignoreFailoverLimit='false', storageDomainId='c8653dba-63ec-4447-872c-0ffcc6af9c0f', imageGroupId='def9e7db-b58b-4df1-ad2b-c47dc60c4303', imageId='357577c1-8d1b-4fad-b470-8e55341e26f6'}), log id: bdcb3c6 2017-10-01 15:20:28,263-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler1) [35bcb02d] -- executeIrsBrokerCommand: calling 'setVolumeDescription', parameters: 2017-10-01 15:20:28,263-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler1) [35bcb02d] ++ spUUID=aca61eac-ed8a-484b-bbee-3645148f6405 2017-10-01 15:20:28,263-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler1) [35bcb02d] ++ sdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f 2017-10-01 15:20:28,263-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler1) [35bcb02d] ++ imageGroupGUID=def9e7db-b58b-4df1-ad2b-c47dc60c4303 2017-10-01 15:20:28,263-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler1) [35bcb02d] ++ volUUID=357577c1-8d1b-4fad-b470-8e55341e26f6 2017-10-01 15:20:28,263-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler1) [35bcb02d] ++ description={"Updated":false,"Last Updated":"Wed Sep 27 10:20:12 EDT 2017","Storage Domains":[{"uuid":"c8653dba-63ec-4447-872c-0ffcc6af9c0f"}],"Disk Description":"OVF_STORE"} 2017-10-01 15:20:28,306-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler1) [35bcb02d] FINISH, SetVolumeDescriptionVDSCommand, log id: bdcb3c6 2017-10-01 15:20:28,320-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.UploadStreamCommand] (DefaultQuartzScheduler1) [35bcb02d] Lock Acquired to object 'EngineLock:{exclusiveLocks='', sharedLocks='[5ae29ed2-9c0f-4754-8df3-b43d14726846=VDS_EXECUTION]'}' 2017-10-01 15:20:28,338-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.UploadStreamCommand] (DefaultQuartzScheduler1) [35bcb02d] Running command: UploadStreamCommand internal: true. Entities affected : ID: c8653dba-63ec-4447-872c-0ffcc6af9c0f Type: Storage 2017-10-01 15:20:28,338-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (DefaultQuartzScheduler1) [35bcb02d] START, UploadStreamVDSCommand(HostName = virt4, UploadStreamVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846'}), log id: 2aef9b24 2017-10-01 15:20:28,338-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (DefaultQuartzScheduler1) [35bcb02d] -- executeVdsBrokerCommand, parameters: 2017-10-01 15:20:28,338-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (DefaultQuartzScheduler1) [35bcb02d] ++ spUUID=aca61eac-ed8a-484b-bbee-3645148f6405 2017-10-01 15:20:28,338-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (DefaultQuartzScheduler1) [35bcb02d] ++ sdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f 2017-10-01 15:20:28,338-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (DefaultQuartzScheduler1) [35bcb02d] ++ imageGUID=def9e7db-b58b-4df1-ad2b-c47dc60c4303 2017-10-01 15:20:28,338-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (DefaultQuartzScheduler1) [35bcb02d] ++ volUUID=357577c1-8d1b-4fad-b470-8e55341e26f6 2017-10-01 15:20:28,338-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (DefaultQuartzScheduler1) [35bcb02d] ++ size=481280 2017-10-01 15:20:28,434-04 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler8) [2b63c427] Polling and updating Async Tasks: 1 tasks, 1 tasks to poll now 2017-10-01 15:20:28,457-04 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler8) [2b63c427] SPMAsyncTask::PollTask: Polling task '0fa9c866-7406-4972-9cab-eea5eaeed8ca' (Parent Command 'UploadStream', Parameters Type 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters') returned status 'finished', result 'success'. 2017-10-01 15:20:28,457-04 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler8) [2b63c427] BaseAsyncTask::onTaskEndSuccess: Task '0fa9c866-7406-4972-9cab-eea5eaeed8ca' (Parent Command 'UploadStream', Parameters Type 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters') ended successfully. 2017-10-01 15:20:28,460-04 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler8) [2b63c427] CommandAsyncTask::endActionIfNecessary: All tasks of command '756c18c8-aded-4699-b45d-42242da0118c' has ended -> executing 'endAction' 2017-10-01 15:20:28,460-04 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler8) [2b63c427] CommandAsyncTask::endAction: Ending action for '1' tasks (command ID: '756c18c8-aded-4699-b45d-42242da0118c'): calling endAction '. 2017-10-01 15:20:28,460-04 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-6-thread-42) [2b63c427] CommandAsyncTask::endCommandAction [within thread] context: Attempting to endAction 'UploadStream', 2017-10-01 15:20:28,466-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.UploadStreamCommand] (org.ovirt.thread.pool-6-thread-42) [35bcb02d] Ending command 'org.ovirt.engine.core.bll.storage.ovfstore.UploadStreamCommand' successfully. 2017-10-01 15:20:28,471-04 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-6-thread-42) [35bcb02d] CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type 'UploadStream' completed, handling the result. 2017-10-01 15:20:28,471-04 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-6-thread-42) [35bcb02d] CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type 'UploadStream' succeeded, clearing tasks. 2017-10-01 15:20:28,471-04 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-6-thread-42) [35bcb02d] SPMAsyncTask::ClearAsyncTask: Attempting to clear task '0fa9c866-7406-4972-9cab-eea5eaeed8ca' 2017-10-01 15:20:28,471-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-6-thread-42) [35bcb02d] START, SPMClearTaskVDSCommand( SPMTaskGuidBaseVDSCommandParameters:{runAsync='true', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', ignoreFailoverLimit='false', taskId='0fa9c866-7406-4972-9cab-eea5eaeed8ca'}), log id: 44bf092c 2017-10-01 15:20:28,471-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-6-thread-42) [35bcb02d] START, HSMClearTaskVDSCommand(HostName = virt4, HSMTaskGuidBaseVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', taskId='0fa9c866-7406-4972-9cab-eea5eaeed8ca'}), log id: 374f0b0d 2017-10-01 15:20:29,460-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (DefaultQuartzScheduler1) [35bcb02d] FINISH, UploadStreamVDSCommand, log id: 2aef9b24 2017-10-01 15:20:29,474-04 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler1) [35bcb02d] CommandAsyncTask::Adding CommandMultiAsyncTasks object for command 'c2d13bb2-3aef-4a36-bdec-79e670002c53' 2017-10-01 15:20:29,474-04 INFO [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (DefaultQuartzScheduler1) [35bcb02d] CommandMultiAsyncTasks::attachTask: Attaching task '63f506c7-30b8-40e2-8ebf-11623a3c72d0' to command 'c2d13bb2-3aef-4a36-bdec-79e670002c53'. 2017-10-01 15:20:29,481-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-6-thread-42) [35bcb02d] FINISH, HSMClearTaskVDSCommand, log id: 374f0b0d 2017-10-01 15:20:29,481-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-6-thread-42) [35bcb02d] FINISH, SPMClearTaskVDSCommand, log id: 44bf092c 2017-10-01 15:20:29,483-04 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-6-thread-42) [35bcb02d] BaseAsyncTask::removeTaskFromDB: Removed task '0fa9c866-7406-4972-9cab-eea5eaeed8ca' from DataBase 2017-10-01 15:20:29,483-04 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-6-thread-42) [35bcb02d] CommandAsyncTask::HandleEndActionResult [within thread]: Removing CommandMultiAsyncTasks object for entity '756c18c8-aded-4699-b45d-42242da0118c' 2017-10-01 15:20:29,486-04 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler1) [35bcb02d] Adding task '63f506c7-30b8-40e2-8ebf-11623a3c72d0' (Parent Command 'UploadStream', Parameters Type 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters'), polling hasn't started yet.. 2017-10-01 15:20:29,492-04 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler1) [35bcb02d] BaseAsyncTask::startPollingTask: Starting to poll task '63f506c7-30b8-40e2-8ebf-11623a3c72d0'. 2017-10-01 15:20:29,496-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.UploadStreamCommand] (DefaultQuartzScheduler1) [35bcb02d] Lock freed to object 'EngineLock:{exclusiveLocks='', sharedLocks='[5ae29ed2-9c0f-4754-8df3-b43d14726846=VDS_EXECUTION]'}' 2017-10-01 15:20:29,497-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler1) [35bcb02d] START, SetVolumeDescriptionVDSCommand( SetVolumeDescriptionVDSCommandParameters:{runAsync='true', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', ignoreFailoverLimit='false', storageDomainId='c8653dba-63ec-4447-872c-0ffcc6af9c0f', imageGroupId='def9e7db-b58b-4df1-ad2b-c47dc60c4303', imageId='357577c1-8d1b-4fad-b470-8e55341e26f6'}), log id: 66a0e75e 2017-10-01 15:20:29,497-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler1) [35bcb02d] -- executeIrsBrokerCommand: calling 'setVolumeDescription', parameters: 2017-10-01 15:20:29,497-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler1) [35bcb02d] ++ spUUID=aca61eac-ed8a-484b-bbee-3645148f6405 2017-10-01 15:20:29,497-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler1) [35bcb02d] ++ sdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f 2017-10-01 15:20:29,497-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler1) [35bcb02d] ++ imageGroupGUID=def9e7db-b58b-4df1-ad2b-c47dc60c4303 2017-10-01 15:20:29,497-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler1) [35bcb02d] ++ volUUID=357577c1-8d1b-4fad-b470-8e55341e26f6 2017-10-01 15:20:29,497-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler1) [35bcb02d] ++ description={"Updated":true,"Size":481280,"Last Updated":"Sun Oct 01 15:20:24 EDT 2017","Storage Domains":[{"uuid":"c8653dba-63ec-4447-872c-0ffcc6af9c0f"}],"Disk Description":"OVF_STORE"} 2017-10-01 15:20:30,520-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler1) [35bcb02d] FINISH, SetVolumeDescriptionVDSCommand, log id: 66a0e75e 2017-10-01 15:20:30,526-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStorageDomainCommand] (DefaultQuartzScheduler1) [35bcb02d] Lock freed to object 'EngineLock:{exclusiveLocks='[c8653dba-63ec-4447-872c-0ffcc6af9c0f=STORAGE]', sharedLocks='[aca61eac-ed8a-484b-bbee-3645148f6405=OVF_UPDATE]'}' 2017-10-01 15:20:31,286-04 INFO [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] (DefaultQuartzScheduler9) [35bcb02d] Command 'ProcessOvfUpdateForStorageDomain' (id: 'd36e4b66-c780-456c-b05f-d60a4bca610a') waiting on child command id: 'c2d13bb2-3aef-4a36-bdec-79e670002c53' type:'UploadStream' to complete 2017-10-01 15:20:38,460-04 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler5) [4b4eda5a] Polling and updating Async Tasks: 2 tasks, 1 tasks to poll now 2017-10-01 15:20:39,320-04 INFO [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] (DefaultQuartzScheduler4) [35bcb02d] Command 'ProcessOvfUpdateForStorageDomain' (id: 'd36e4b66-c780-456c-b05f-d60a4bca610a') waiting on child command id: 'c2d13bb2-3aef-4a36-bdec-79e670002c53' type:'UploadStream' to complete 2017-10-01 15:20:39,465-04 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler5) [4b4eda5a] SPMAsyncTask::PollTask: Polling task '63f506c7-30b8-40e2-8ebf-11623a3c72d0' (Parent Command 'UploadStream', Parameters Type 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters') returned status 'finished', result 'success'. 2017-10-01 15:20:39,465-04 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler5) [4b4eda5a] BaseAsyncTask::onTaskEndSuccess: Task '63f506c7-30b8-40e2-8ebf-11623a3c72d0' (Parent Command 'UploadStream', Parameters Type 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters') ended successfully. 2017-10-01 15:20:39,469-04 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler5) [4b4eda5a] CommandAsyncTask::endActionIfNecessary: All tasks of command 'c2d13bb2-3aef-4a36-bdec-79e670002c53' has ended -> executing 'endAction' 2017-10-01 15:20:39,469-04 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler5) [4b4eda5a] CommandAsyncTask::endAction: Ending action for '1' tasks (command ID: 'c2d13bb2-3aef-4a36-bdec-79e670002c53'): calling endAction '. 2017-10-01 15:20:39,470-04 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-6-thread-31) [4b4eda5a] CommandAsyncTask::endCommandAction [within thread] context: Attempting to endAction 'UploadStream', 2017-10-01 15:20:39,475-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.UploadStreamCommand] (org.ovirt.thread.pool-6-thread-31) [35bcb02d] Ending command 'org.ovirt.engine.core.bll.storage.ovfstore.UploadStreamCommand' successfully. 2017-10-01 15:20:39,478-04 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-6-thread-31) [35bcb02d] CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type 'UploadStream' completed, handling the result. 2017-10-01 15:20:39,478-04 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-6-thread-31) [35bcb02d] CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type 'UploadStream' succeeded, clearing tasks. 2017-10-01 15:20:39,478-04 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-6-thread-31) [35bcb02d] SPMAsyncTask::ClearAsyncTask: Attempting to clear task '63f506c7-30b8-40e2-8ebf-11623a3c72d0' 2017-10-01 15:20:39,479-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-6-thread-31) [35bcb02d] START, SPMClearTaskVDSCommand( SPMTaskGuidBaseVDSCommandParameters:{runAsync='true', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', ignoreFailoverLimit='false', taskId='63f506c7-30b8-40e2-8ebf-11623a3c72d0'}), log id: 78f46780 2017-10-01 15:20:39,480-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-6-thread-31) [35bcb02d] START, HSMClearTaskVDSCommand(HostName = virt4, HSMTaskGuidBaseVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', taskId='63f506c7-30b8-40e2-8ebf-11623a3c72d0'}), log id: 1aeb88ea 2017-10-01 15:20:39,959-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-6-thread-31) [35bcb02d] FINISH, HSMClearTaskVDSCommand, log id: 1aeb88ea 2017-10-01 15:20:39,959-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-6-thread-31) [35bcb02d] FINISH, SPMClearTaskVDSCommand, log id: 78f46780 2017-10-01 15:20:39,961-04 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-6-thread-31) [35bcb02d] BaseAsyncTask::removeTaskFromDB: Removed task '63f506c7-30b8-40e2-8ebf-11623a3c72d0' from DataBase 2017-10-01 15:20:39,961-04 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-6-thread-31) [35bcb02d] CommandAsyncTask::HandleEndActionResult [within thread]: Removing CommandMultiAsyncTasks object for entity 'c2d13bb2-3aef-4a36-bdec-79e670002c53' 2017-10-01 15:20:49,356-04 INFO [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] (DefaultQuartzScheduler9) [35bcb02d] Command 'ProcessOvfUpdateForStorageDomain' id: 'd36e4b66-c780-456c-b05f-d60a4bca610a' child commands '[756c18c8-aded-4699-b45d-42242da0118c, c2d13bb2-3aef-4a36-bdec-79e670002c53]' executions were completed, status 'SUCCEEDED' 2017-10-01 15:20:50,383-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStorageDomainCommand] (DefaultQuartzScheduler8) [35bcb02d] Ending command 'org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStorageDomainCommand' successfully. 2017-10-01 15:20:55,398-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [4db14dd8] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 15:21:29,158-04 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler3) [3a1f56ab] Setting new tasks map. The map contains now 1 tasks 2017-10-01 15:21:55,418-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler7) [3a20071a] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 15:21:59,160-04 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler4) [4ad2334e] Setting new tasks map. The map contains now 0 tasks 2017-10-01 15:21:59,160-04 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler4) [4ad2334e] Cleared all tasks of pool 'aca61eac-ed8a-484b-bbee-3645148f6405'. 2017-10-01 15:22:55,444-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [78683017] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 15:23:04,463-04 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-9) [] domain 'f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2:export' in problem 'PROBLEMATIC'. vds: 'virt2' 2017-10-01 15:23:04,463-04 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-9) [] domain 'c8653dba-63ec-4447-872c-0ffcc6af9c0f:data' in problem 'PROBLEMATIC'. vds: 'virt2' 2017-10-01 15:23:04,464-04 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-9) [] domain '63d680c7-4c94-4b29-a6cb-f288a128b9bc:iso' in problem 'PROBLEMATIC'. vds: 'virt2' 2017-10-01 15:23:13,955-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler3) [41c6461a] VM '1734fed5-fc97-4714-bb63-55e8f4eb6d34'(wiki) moved from 'Up' --> 'NotResponding' 2017-10-01 15:23:13,963-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler7) [27abe203] VM '07331c78-6e09-41b4-9b29-edda32f46351'(samba) moved from 'Up' --> 'NotResponding' 2017-10-01 15:23:13,991-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [27abe203] EVENT_ID: VM_NOT_RESPONDING(126), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM samba is not responding. 2017-10-01 15:23:13,992-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler7) [27abe203] VM 'dc66c051-24b7-4cb8-99eb-db398d396ef1'(samba1) moved from 'Up' --> 'NotResponding' 2017-10-01 15:23:14,023-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [27abe203] EVENT_ID: VM_NOT_RESPONDING(126), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM samba1 is not responding. 2017-10-01 15:23:14,024-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler7) [27abe203] VM 'bcd5776e-978b-4723-933d-70a80322a796'(cape) moved from 'Up' --> 'NotResponding' 2017-10-01 15:23:14,031-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [41c6461a] EVENT_ID: VM_NOT_RESPONDING(126), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM wiki is not responding. 2017-10-01 15:23:14,032-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler3) [41c6461a] VM 'cc1dbe93-bb80-451e-8b3b-c9cc5e90432f'(license) moved from 'Up' --> 'NotResponding' 2017-10-01 15:23:14,045-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [27abe203] EVENT_ID: VM_NOT_RESPONDING(126), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM cape is not responding. 2017-10-01 15:23:14,045-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler7) [27abe203] VM '3e5a5728-8940-4fe7-b4af-11fb8e12f116'(www) moved from 'Up' --> 'NotResponding' 2017-10-01 15:23:14,058-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [41c6461a] EVENT_ID: VM_NOT_RESPONDING(126), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM license is not responding. 2017-10-01 15:23:14,058-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler3) [41c6461a] VM '64734fca-4d6c-443c-82d1-abf7cf106601'(webdav) moved from 'Up' --> 'NotResponding' 2017-10-01 15:23:14,066-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [27abe203] EVENT_ID: VM_NOT_RESPONDING(126), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM www is not responding. 2017-10-01 15:23:14,066-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler7) [27abe203] VM 'f87ec08d-0fdf-42ba-8aac-c66c404d7aae'(webapp) moved from 'Up' --> 'NotResponding' 2017-10-01 15:23:14,081-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [41c6461a] EVENT_ID: VM_NOT_RESPONDING(126), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM webdav is not responding. 2017-10-01 15:23:14,088-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [27abe203] EVENT_ID: VM_NOT_RESPONDING(126), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM webapp is not responding. 2017-10-01 15:23:14,089-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler7) [27abe203] VM '911d6883-607c-464e-b46b-8e6141bdd2de'(hdb) moved from 'Up' --> 'NotResponding' 2017-10-01 15:23:14,109-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [27abe203] EVENT_ID: VM_NOT_RESPONDING(126), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM hdb is not responding. 2017-10-01 15:23:14,110-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler7) [27abe203] VM '8391f9a7-7fe7-4025-8fba-6273ae587a52'(archive) moved from 'Up' --> 'NotResponding' 2017-10-01 15:23:14,130-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [27abe203] EVENT_ID: VM_NOT_RESPONDING(126), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM archive is not responding. 2017-10-01 15:23:14,130-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler7) [27abe203] VM '1c7e966f-29e8-4788-b64c-86e52f415061'(udb) moved from 'Up' --> 'NotResponding' 2017-10-01 15:23:14,150-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [27abe203] EVENT_ID: VM_NOT_RESPONDING(126), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM udb is not responding. 2017-10-01 15:23:14,150-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler7) [27abe203] VM '9ea2821e-557f-45cb-9872-1e6c5aeaf246'(rs) moved from 'Up' --> 'NotResponding' 2017-10-01 15:23:14,170-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [27abe203] EVENT_ID: VM_NOT_RESPONDING(126), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM rs is not responding. 2017-10-01 15:23:14,171-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler7) [27abe203] VM 'd66235b4-7c8d-4da2-a572-82ecc9c53a59'(www-las) moved from 'Up' --> 'NotResponding' 2017-10-01 15:23:14,191-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [27abe203] EVENT_ID: VM_NOT_RESPONDING(126), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM www-las is not responding. 2017-10-01 15:23:14,191-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler7) [27abe203] VM '5885c43a-b133-45dc-b8d5-1aa2a4a19579'(prs) moved from 'Up' --> 'NotResponding' 2017-10-01 15:23:14,211-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [27abe203] EVENT_ID: VM_NOT_RESPONDING(126), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM prs is not responding. 2017-10-01 15:23:14,211-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler7) [27abe203] VM 'ae65a104-ed3f-40e0-ba92-5708cf338918'(vpn2) moved from 'Up' --> 'NotResponding' 2017-10-01 15:23:14,231-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [27abe203] EVENT_ID: VM_NOT_RESPONDING(126), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM vpn2 is not responding. 2017-10-01 15:23:14,231-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler7) [27abe203] VM 'd56363ad-9cb1-4a70-a33a-ce439b044d7d'(svn) moved from 'Up' --> 'NotResponding' 2017-10-01 15:23:14,251-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [27abe203] EVENT_ID: VM_NOT_RESPONDING(126), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM svn is not responding. 2017-10-01 15:23:25,320-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler6) [481096c7] VM 'db3e08fb-c111-442d-aeec-1507da30f486'(print) moved from 'Up' --> 'NotResponding' 2017-10-01 15:23:25,342-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [481096c7] EVENT_ID: VM_NOT_RESPONDING(126), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM print is not responding. 2017-10-01 15:23:25,342-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler6) [481096c7] VM 'ae56c31c-71f7-433a-b198-a1c2c846325e'(forum) moved from 'Up' --> 'NotResponding' 2017-10-01 15:23:25,363-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [481096c7] EVENT_ID: VM_NOT_RESPONDING(126), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM forum is not responding. 2017-10-01 15:23:25,363-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler6) [481096c7] VM 'ddf37076-e795-4c87-a4fb-9efcaae76d72'(dc1) moved from 'Up' --> 'NotResponding' 2017-10-01 15:23:25,384-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [481096c7] EVENT_ID: VM_NOT_RESPONDING(126), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM dc1 is not responding. 2017-10-01 15:23:25,384-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler6) [481096c7] VM 'bfe9a524-2238-4bab-a91f-5dc6ed270896'(pcmode) moved from 'Up' --> 'NotResponding' 2017-10-01 15:23:25,404-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [481096c7] EVENT_ID: VM_NOT_RESPONDING(126), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM pcmode is not responding. 2017-10-01 15:23:25,405-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler6) [481096c7] VM '1325cebf-834e-4571-a3e1-82beb0021ffe'(dns1) moved from 'Up' --> 'NotResponding' 2017-10-01 15:23:25,425-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [481096c7] EVENT_ID: VM_NOT_RESPONDING(126), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM dns1 is not responding. 2017-10-01 15:23:25,425-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler6) [481096c7] VM 'a82b4cc3-6ef6-4a8e-97ef-1d4ea1b7e07c'(indigo) moved from 'Up' --> 'NotResponding' 2017-10-01 15:23:25,445-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [481096c7] EVENT_ID: VM_NOT_RESPONDING(126), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM indigo is not responding. 2017-10-01 15:23:25,446-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler6) [481096c7] VM 'c8682859-9e55-460a-9ef0-d03e4a85f50e'(dhcp1) moved from 'Up' --> 'NotResponding' 2017-10-01 15:23:25,466-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [481096c7] EVENT_ID: VM_NOT_RESPONDING(126), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM dhcp1 is not responding. 2017-10-01 15:23:25,466-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler6) [481096c7] VM '3bb83ad5-dc00-445d-8708-2db18c53e0e6'(www-vhost) moved from 'Up' --> 'NotResponding' 2017-10-01 15:23:25,487-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [481096c7] EVENT_ID: VM_NOT_RESPONDING(126), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM www-vhost is not responding. 2017-10-01 15:23:25,487-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler6) [481096c7] VM 'a8d4e149-55ec-4961-a414-d7d1e1b05a59'(fix) moved from 'Up' --> 'NotResponding' 2017-10-01 15:23:25,507-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [481096c7] EVENT_ID: VM_NOT_RESPONDING(126), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM fix is not responding. 2017-10-01 15:23:25,507-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler6) [481096c7] VM 'aa487207-7ff4-465a-9d9b-2a103d50dc77'(ftp1) moved from 'Up' --> 'NotResponding' 2017-10-01 15:23:25,528-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [481096c7] EVENT_ID: VM_NOT_RESPONDING(126), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM ftp1 is not responding. 2017-10-01 15:23:25,528-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler6) [481096c7] VM '4ba80442-b361-4026-ae8c-895fd28042b0'(dist) moved from 'Up' --> 'NotResponding' 2017-10-01 15:23:25,549-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [481096c7] EVENT_ID: VM_NOT_RESPONDING(126), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM dist is not responding. 2017-10-01 15:23:40,322-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler4) [2526f1fb] VM '899e3a1c-dcc4-4426-9dd4-0f1e8b94a2b8'(ipa) moved from 'Up' --> 'NotResponding' 2017-10-01 15:23:40,393-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [2526f1fb] EVENT_ID: VM_NOT_RESPONDING(126), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM ipa is not responding. 2017-10-01 15:23:40,393-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler4) [2526f1fb] VM '77766cb4-0625-4432-a16f-def5e702102a'(labtest1) moved from 'Up' --> 'NotResponding' 2017-10-01 15:23:40,415-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [2526f1fb] EVENT_ID: VM_NOT_RESPONDING(126), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM labtest1 is not responding. 2017-10-01 15:23:40,415-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler4) [2526f1fb] VM '59287796-c634-4f3c-94b1-2c9699986637'(vpn1) moved from 'Up' --> 'NotResponding' 2017-10-01 15:23:40,435-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [2526f1fb] EVENT_ID: VM_NOT_RESPONDING(126), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM vpn1 is not responding. 2017-10-01 15:23:47,148-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler1) [45199903] Command 'GetAllVmStatsVDSCommand(HostName = virt4, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846'})' execution failed: VDSGenericException: VDSNetworkException: Heartbeat exceeded 2017-10-01 15:23:47,156-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.PollVmStatsRefresher] (DefaultQuartzScheduler1) [45199903] Failed to fetch vms info for host 'virt4' - skipping VMs monitoring. 2017-10-01 15:23:47,161-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [41bf5df1] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command GetStatsVDS failed: Heartbeat exceeded 2017-10-01 15:23:47,161-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStatsVDSCommand] (DefaultQuartzScheduler2) [41bf5df1] Command 'GetStatsVDSCommand(HostName = virt4, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vds='Host[virt4,5ae29ed2-9c0f-4754-8df3-b43d14726846]'})' execution failed: VDSGenericException: VDSNetworkException: Heartbeat exceeded 2017-10-01 15:23:47,161-04 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [41bf5df1] Failed getting vds stats, host='virt4'(5ae29ed2-9c0f-4754-8df3-b43d14726846): org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Heartbeat exceeded 2017-10-01 15:23:47,161-04 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [41bf5df1] Failure to refresh host 'virt4' runtime info: VDSGenericException: VDSNetworkException: Heartbeat exceeded 2017-10-01 15:23:47,161-04 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler2) [41bf5df1] Failed to refresh VDS, network error, continuing, vds='virt4'(5ae29ed2-9c0f-4754-8df3-b43d14726846): VDSGenericException: VDSNetworkException: Heartbeat exceeded 2017-10-01 15:23:47,161-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [783821c3] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command SpmStatusVDS failed: Heartbeat exceeded 2017-10-01 15:23:47,161-04 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (org.ovirt.thread.pool-6-thread-1) [41bf5df1] Host 'virt4' is not responding. It will stay in Connecting state for a grace period of 86 seconds and after that an attempt to fence the host will be issued. 2017-10-01 15:23:47,161-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler5) [783821c3] Command 'SpmStatusVDSCommand(HostName = virt4, SpmStatusVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: VDSGenericException: VDSNetworkException: Heartbeat exceeded 2017-10-01 15:23:47,161-04 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (org.ovirt.thread.pool-6-thread-19) [45199903] Host 'virt4' is not responding. It will stay in Connecting state for a grace period of 86 seconds and after that an attempt to fence the host will be issued. 2017-10-01 15:23:47,166-04 INFO [org.ovirt.engine.core.bll.storage.pool.SetStoragePoolStatusCommand] (DefaultQuartzScheduler5) [5532121c] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: aca61eac-ed8a-484b-bbee-3645148f6405 Type: StoragePool 2017-10-01 15:23:47,167-04 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler5) [5532121c] Storage Pool 'aca61eac-ed8a-484b-bbee-3645148f6405' - Updating Storage Domain '63d680c7-4c94-4b29-a6cb-f288a128b9bc' status from 'Active' to 'Unknown', reason: null 2017-10-01 15:23:47,168-04 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler5) [5532121c] Storage Pool 'aca61eac-ed8a-484b-bbee-3645148f6405' - Updating Storage Domain 'f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2' status from 'Active' to 'Unknown', reason: null 2017-10-01 15:23:47,168-04 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler5) [5532121c] Storage Pool 'aca61eac-ed8a-484b-bbee-3645148f6405' - Updating Storage Domain 'c8653dba-63ec-4447-872c-0ffcc6af9c0f' status from 'Active' to 'Unknown', reason: null 2017-10-01 15:23:47,194-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-1) [41bf5df1] EVENT_ID: VDS_HOST_NOT_RESPONDING_CONNECTING(9,008), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host virt4 is not responding. It will stay in Connecting state for a grace period of 86 seconds and after that an attempt to fence the host will be issued. 2017-10-01 15:23:47,194-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-19) [45199903] EVENT_ID: VDS_HOST_NOT_RESPONDING_CONNECTING(9,008), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host virt4 is not responding. It will stay in Connecting state for a grace period of 86 seconds and after that an attempt to fence the host will be issued. 2017-10-01 15:23:47,215-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [5532121c] EVENT_ID: SYSTEM_CHANGE_STORAGE_POOL_STATUS_PROBLEMATIC_WITH_ERROR(987), Correlation ID: 5532121c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Invalid status on Data Center EECS-Primary. Setting Data Center status to Non Responsive (On host virt4, Error: Network error during communication with the Host.). 2017-10-01 15:23:47,316-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [5532121c] EVENT_ID: IRS_BROKER_COMMAND_FAILURE(10,803), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM command GetStoragePoolInfoVDS failed: Heartbeat exceeded 2017-10-01 15:23:47,317-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler5) [5532121c] ERROR, org.ovirt.engine.core.vdsbroker.irsbroker.GetStoragePoolInfoVDSCommand, exception: VDSGenericException: VDSNetworkException: Heartbeat exceeded, log id: 3b2d18f 2017-10-01 15:23:47,317-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler5) [5532121c] Exception 2017-10-01 15:23:47,365-04 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to virt4-mgmt/192.168.0.37 2017-10-01 15:23:47,367-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler5) [5532121c] Command 'SpmStatusVDSCommand(HostName = virt4, SpmStatusVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: java.net.ConnectException: Connection refused 2017-10-01 15:23:47,367-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler5) [5532121c] Irs placed on server '5ae29ed2-9c0f-4754-8df3-b43d14726846' failed. Proceed Failover 2017-10-01 15:23:47,423-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [5532121c] START, ConnectStoragePoolVDSCommand(HostName = virt1, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vdsId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 5605c5bd 2017-10-01 15:23:47,423-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [5532121c] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 15:23:50,177-04 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to virt4-mgmt/192.168.0.37 2017-10-01 15:23:50,177-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler2) [26e0cd11] Command 'GetCapabilitiesVDSCommand(HostName = virt4, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vds='Host[virt4,5ae29ed2-9c0f-4754-8df3-b43d14726846]'})' execution failed: java.net.ConnectException: Connection refused 2017-10-01 15:23:50,177-04 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [26e0cd11] Failure to refresh host 'virt4' runtime info: java.net.ConnectException: Connection refused 2017-10-01 15:23:50,177-04 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler2) [26e0cd11] Failed to refresh VDS, network error, continuing, vds='virt4'(5ae29ed2-9c0f-4754-8df3-b43d14726846): java.net.ConnectException: Connection refused 2017-10-01 15:23:53,192-04 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to virt4-mgmt/192.168.0.37 2017-10-01 15:23:53,193-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler10) [340b4094] Command 'GetCapabilitiesVDSCommand(HostName = virt4, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vds='Host[virt4,5ae29ed2-9c0f-4754-8df3-b43d14726846]'})' execution failed: java.net.ConnectException: Connection refused 2017-10-01 15:23:53,195-04 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [340b4094] Failure to refresh host 'virt4' runtime info: java.net.ConnectException: Connection refused 2017-10-01 15:23:53,195-04 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler10) [340b4094] Failed to refresh VDS, network error, continuing, vds='virt4'(5ae29ed2-9c0f-4754-8df3-b43d14726846): java.net.ConnectException: Connection refused 2017-10-01 15:23:55,464-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler7) [217a845d] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 15:23:56,210-04 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to virt4-mgmt/192.168.0.37 2017-10-01 15:23:56,211-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler6) [235c756a] Command 'GetCapabilitiesVDSCommand(HostName = virt4, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vds='Host[virt4,5ae29ed2-9c0f-4754-8df3-b43d14726846]'})' execution failed: java.net.ConnectException: Connection refused 2017-10-01 15:23:56,211-04 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [235c756a] Failure to refresh host 'virt4' runtime info: java.net.ConnectException: Connection refused 2017-10-01 15:23:56,211-04 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler6) [235c756a] Failed to refresh VDS, network error, continuing, vds='virt4'(5ae29ed2-9c0f-4754-8df3-b43d14726846): java.net.ConnectException: Connection refused 2017-10-01 15:23:59,226-04 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to virt4-mgmt/192.168.0.37 2017-10-01 15:23:59,227-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler6) [7b544677] Command 'GetCapabilitiesVDSCommand(HostName = virt4, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vds='Host[virt4,5ae29ed2-9c0f-4754-8df3-b43d14726846]'})' execution failed: java.net.ConnectException: Connection refused 2017-10-01 15:23:59,227-04 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [7b544677] Failure to refresh host 'virt4' runtime info: java.net.ConnectException: Connection refused 2017-10-01 15:23:59,227-04 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler6) [7b544677] Failed to refresh VDS, network error, continuing, vds='virt4'(5ae29ed2-9c0f-4754-8df3-b43d14726846): java.net.ConnectException: Connection refused 2017-10-01 15:24:02,157-04 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to virt4-mgmt/192.168.0.37 2017-10-01 15:24:02,158-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler1) [39ebb7b5] Command 'GetAllVmStatsVDSCommand(HostName = virt4, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846'})' execution failed: java.net.ConnectException: Connection refused 2017-10-01 15:24:02,158-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.PollVmStatsRefresher] (DefaultQuartzScheduler1) [39ebb7b5] Failed to fetch vms info for host 'virt4' - skipping VMs monitoring. 2017-10-01 15:24:02,242-04 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to virt4-mgmt/192.168.0.37 2017-10-01 15:24:02,243-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler8) [214c01e] Command 'GetCapabilitiesVDSCommand(HostName = virt4, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vds='Host[virt4,5ae29ed2-9c0f-4754-8df3-b43d14726846]'})' execution failed: java.net.ConnectException: Connection refused 2017-10-01 15:24:02,243-04 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [214c01e] Failure to refresh host 'virt4' runtime info: java.net.ConnectException: Connection refused 2017-10-01 15:24:02,243-04 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler8) [214c01e] Failed to refresh VDS, network error, continuing, vds='virt4'(5ae29ed2-9c0f-4754-8df3-b43d14726846): java.net.ConnectException: Connection refused 2017-10-01 15:24:05,258-04 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to virt4-mgmt/192.168.0.37 2017-10-01 15:24:05,259-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler2) [2308919a] Command 'GetCapabilitiesVDSCommand(HostName = virt4, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vds='Host[virt4,5ae29ed2-9c0f-4754-8df3-b43d14726846]'})' execution failed: java.net.ConnectException: Connection refused 2017-10-01 15:24:05,259-04 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [2308919a] Failure to refresh host 'virt4' runtime info: java.net.ConnectException: Connection refused 2017-10-01 15:24:05,259-04 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler2) [2308919a] Failed to refresh VDS, network error, continuing, vds='virt4'(5ae29ed2-9c0f-4754-8df3-b43d14726846): java.net.ConnectException: Connection refused 2017-10-01 15:24:08,274-04 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to virt4-mgmt/192.168.0.37 2017-10-01 15:24:08,275-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler3) [2c3509ca] Command 'GetCapabilitiesVDSCommand(HostName = virt4, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vds='Host[virt4,5ae29ed2-9c0f-4754-8df3-b43d14726846]'})' execution failed: java.net.ConnectException: Connection refused 2017-10-01 15:24:08,275-04 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [2c3509ca] Failure to refresh host 'virt4' runtime info: java.net.ConnectException: Connection refused 2017-10-01 15:24:08,275-04 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler3) [2c3509ca] Failed to refresh VDS, network error, continuing, vds='virt4'(5ae29ed2-9c0f-4754-8df3-b43d14726846): java.net.ConnectException: Connection refused 2017-10-01 15:24:11,290-04 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to virt4-mgmt/192.168.0.37 2017-10-01 15:24:11,290-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler6) [7b36009b] Command 'GetCapabilitiesVDSCommand(HostName = virt4, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vds='Host[virt4,5ae29ed2-9c0f-4754-8df3-b43d14726846]'})' execution failed: java.net.ConnectException: Connection refused 2017-10-01 15:24:11,290-04 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [7b36009b] Failure to refresh host 'virt4' runtime info: java.net.ConnectException: Connection refused 2017-10-01 15:24:11,290-04 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler6) [7b36009b] Failed to refresh VDS, network error, continuing, vds='virt4'(5ae29ed2-9c0f-4754-8df3-b43d14726846): java.net.ConnectException: Connection refused 2017-10-01 15:24:14,305-04 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to virt4-mgmt/192.168.0.37 2017-10-01 15:24:14,305-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler1) [4580d4ab] Command 'GetCapabilitiesVDSCommand(HostName = virt4, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vds='Host[virt4,5ae29ed2-9c0f-4754-8df3-b43d14726846]'})' execution failed: java.net.ConnectException: Connection refused 2017-10-01 15:24:14,305-04 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [4580d4ab] Failure to refresh host 'virt4' runtime info: java.net.ConnectException: Connection refused 2017-10-01 15:24:14,305-04 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler1) [4580d4ab] Failed to refresh VDS, network error, continuing, vds='virt4'(5ae29ed2-9c0f-4754-8df3-b43d14726846): java.net.ConnectException: Connection refused 2017-10-01 15:24:17,159-04 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to virt4-mgmt/192.168.0.37 2017-10-01 15:24:17,160-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler10) [730e36af] Command 'GetAllVmStatsVDSCommand(HostName = virt4, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846'})' execution failed: java.net.ConnectException: Connection refused 2017-10-01 15:24:17,160-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.PollVmStatsRefresher] (DefaultQuartzScheduler10) [730e36af] Failed to fetch vms info for host 'virt4' - skipping VMs monitoring. 2017-10-01 15:24:17,320-04 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to virt4-mgmt/192.168.0.37 2017-10-01 15:24:17,321-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler9) [cccd7df] Command 'GetCapabilitiesVDSCommand(HostName = virt4, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vds='Host[virt4,5ae29ed2-9c0f-4754-8df3-b43d14726846]'})' execution failed: java.net.ConnectException: Connection refused 2017-10-01 15:24:17,321-04 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [cccd7df] Failure to refresh host 'virt4' runtime info: java.net.ConnectException: Connection refused 2017-10-01 15:24:17,321-04 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler9) [cccd7df] Failed to refresh VDS, network error, continuing, vds='virt4'(5ae29ed2-9c0f-4754-8df3-b43d14726846): java.net.ConnectException: Connection refused 2017-10-01 15:24:20,336-04 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to virt4-mgmt/192.168.0.37 2017-10-01 15:24:20,337-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler8) [3b328b8f] Command 'GetCapabilitiesVDSCommand(HostName = virt4, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vds='Host[virt4,5ae29ed2-9c0f-4754-8df3-b43d14726846]'})' execution failed: java.net.ConnectException: Connection refused 2017-10-01 15:24:20,337-04 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [3b328b8f] Failure to refresh host 'virt4' runtime info: java.net.ConnectException: Connection refused 2017-10-01 15:24:20,337-04 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler8) [3b328b8f] Failed to refresh VDS, network error, continuing, vds='virt4'(5ae29ed2-9c0f-4754-8df3-b43d14726846): java.net.ConnectException: Connection refused 2017-10-01 15:24:23,352-04 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to virt4-mgmt/192.168.0.37 2017-10-01 15:24:23,353-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler7) [9d8ac6f] Command 'GetCapabilitiesVDSCommand(HostName = virt4, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vds='Host[virt4,5ae29ed2-9c0f-4754-8df3-b43d14726846]'})' execution failed: java.net.ConnectException: Connection refused 2017-10-01 15:24:23,353-04 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [9d8ac6f] Failure to refresh host 'virt4' runtime info: java.net.ConnectException: Connection refused 2017-10-01 15:24:23,353-04 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler7) [9d8ac6f] Failed to refresh VDS, network error, continuing, vds='virt4'(5ae29ed2-9c0f-4754-8df3-b43d14726846): java.net.ConnectException: Connection refused 2017-10-01 15:24:26,368-04 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to virt4-mgmt/192.168.0.37 2017-10-01 15:24:26,369-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler6) [769052f] Command 'GetCapabilitiesVDSCommand(HostName = virt4, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vds='Host[virt4,5ae29ed2-9c0f-4754-8df3-b43d14726846]'})' execution failed: java.net.ConnectException: Connection refused 2017-10-01 15:24:26,369-04 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [769052f] Failure to refresh host 'virt4' runtime info: java.net.ConnectException: Connection refused 2017-10-01 15:24:26,369-04 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler6) [769052f] Failed to refresh VDS, network error, continuing, vds='virt4'(5ae29ed2-9c0f-4754-8df3-b43d14726846): java.net.ConnectException: Connection refused 2017-10-01 15:24:29,384-04 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to virt4-mgmt/192.168.0.37 2017-10-01 15:24:29,385-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler2) [247cafd7] Command 'GetCapabilitiesVDSCommand(HostName = virt4, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vds='Host[virt4,5ae29ed2-9c0f-4754-8df3-b43d14726846]'})' execution failed: java.net.ConnectException: Connection refused 2017-10-01 15:24:29,385-04 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [247cafd7] Failure to refresh host 'virt4' runtime info: java.net.ConnectException: Connection refused 2017-10-01 15:24:29,385-04 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler2) [247cafd7] Failed to refresh VDS, network error, continuing, vds='virt4'(5ae29ed2-9c0f-4754-8df3-b43d14726846): java.net.ConnectException: Connection refused 2017-10-01 15:24:32,159-04 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to virt4-mgmt/192.168.0.37 2017-10-01 15:24:32,165-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler3) [31386eb7] Command 'GetAllVmStatsVDSCommand(HostName = virt4, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846'})' execution failed: java.net.ConnectException: Connection refused 2017-10-01 15:24:32,165-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.PollVmStatsRefresher] (DefaultQuartzScheduler3) [31386eb7] Failed to fetch vms info for host 'virt4' - skipping VMs monitoring. 2017-10-01 15:24:32,431-04 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to virt4-mgmt/192.168.0.37 2017-10-01 15:24:32,431-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler7) [385d668b] Command 'GetCapabilitiesVDSCommand(HostName = virt4, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vds='Host[virt4,5ae29ed2-9c0f-4754-8df3-b43d14726846]'})' execution failed: java.net.ConnectException: Connection refused 2017-10-01 15:24:32,431-04 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [385d668b] Failure to refresh host 'virt4' runtime info: java.net.ConnectException: Connection refused 2017-10-01 15:24:32,432-04 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler7) [385d668b] Failed to refresh VDS, network error, continuing, vds='virt4'(5ae29ed2-9c0f-4754-8df3-b43d14726846): java.net.ConnectException: Connection refused 2017-10-01 15:24:35,447-04 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to virt4-mgmt/192.168.0.37 2017-10-01 15:24:35,585-04 ERROR [org.ovirt.vdsm.jsonrpc.client.JsonRpcClient] (ResponseWorker) [] Not able to update response for "8809df95-8daf-408f-b062-5505d601d962" 2017-10-01 15:24:36,468-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [4c611ebc] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command GetCapabilitiesVDS failed: Recovering from crash or Initializing 2017-10-01 15:24:36,468-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler4) [4c611ebc] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@6bc09ead' 2017-10-01 15:24:36,468-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler4) [4c611ebc] HostName = virt4 2017-10-01 15:24:36,468-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler4) [4c611ebc] Command 'GetCapabilitiesVDSCommand(HostName = virt4, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vds='Host[virt4,5ae29ed2-9c0f-4754-8df3-b43d14726846]'})' execution failed: Recovering from crash or Initializing 2017-10-01 15:24:36,487-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [4c611ebc] EVENT_ID: VDS_INITIALIZING(514), Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSRecoveringException: Recovering from crash or Initializing 2017-10-01 15:24:36,487-04 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler4) [4c611ebc] Failed to refresh VDS, continuing, vds='virt4'(5ae29ed2-9c0f-4754-8df3-b43d14726846): Recovering from crash or Initializing 2017-10-01 15:24:40,509-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [69f6f379] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command GetCapabilitiesVDS failed: Recovering from crash or Initializing 2017-10-01 15:24:40,509-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler6) [69f6f379] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@33bea50f' 2017-10-01 15:24:40,509-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler6) [69f6f379] HostName = virt4 2017-10-01 15:24:40,509-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler6) [69f6f379] Command 'GetCapabilitiesVDSCommand(HostName = virt4, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vds='Host[virt4,5ae29ed2-9c0f-4754-8df3-b43d14726846]'})' execution failed: Recovering from crash or Initializing 2017-10-01 15:24:44,531-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [1569c8c4] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command GetCapabilitiesVDS failed: Recovering from crash or Initializing 2017-10-01 15:24:44,531-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler2) [1569c8c4] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@c202f55' 2017-10-01 15:24:44,531-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler2) [1569c8c4] HostName = virt4 2017-10-01 15:24:44,531-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler2) [1569c8c4] Command 'GetCapabilitiesVDSCommand(HostName = virt4, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vds='Host[virt4,5ae29ed2-9c0f-4754-8df3-b43d14726846]'})' execution failed: Recovering from crash or Initializing 2017-10-01 15:24:47,169-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler8) [6d5f844e] Command 'GetAllVmStatsVDSCommand(HostName = virt4, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846'})' execution failed: Recovering from crash or Initializing 2017-10-01 15:24:47,169-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.PollVmStatsRefresher] (DefaultQuartzScheduler8) [6d5f844e] Failed to fetch vms info for host 'virt4' - skipping VMs monitoring. 2017-10-01 15:24:48,554-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [5d8ba1ab] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command GetCapabilitiesVDS failed: Recovering from crash or Initializing 2017-10-01 15:24:48,554-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler10) [5d8ba1ab] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@721b317a' 2017-10-01 15:24:48,554-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler10) [5d8ba1ab] HostName = virt4 2017-10-01 15:24:48,554-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler10) [5d8ba1ab] Command 'GetCapabilitiesVDSCommand(HostName = virt4, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vds='Host[virt4,5ae29ed2-9c0f-4754-8df3-b43d14726846]'})' execution failed: Recovering from crash or Initializing 2017-10-01 15:24:52,559-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [1fbab35c] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command GetCapabilitiesVDS failed: Recovering from crash or Initializing 2017-10-01 15:24:52,559-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler4) [1fbab35c] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@fb8d6fb' 2017-10-01 15:24:52,559-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler4) [1fbab35c] HostName = virt4 2017-10-01 15:24:52,559-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler4) [1fbab35c] Command 'GetCapabilitiesVDSCommand(HostName = virt4, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vds='Host[virt4,5ae29ed2-9c0f-4754-8df3-b43d14726846]'})' execution failed: Recovering from crash or Initializing 2017-10-01 15:24:55,483-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [176c59a7] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 15:24:55,671-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [5fe4e2a4] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command GetCapabilitiesVDS failed: Recovering from crash or Initializing 2017-10-01 15:24:55,671-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler2) [5fe4e2a4] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@5fedc92c' 2017-10-01 15:24:55,671-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler2) [5fe4e2a4] HostName = virt4 2017-10-01 15:24:55,671-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler2) [5fe4e2a4] Command 'GetCapabilitiesVDSCommand(HostName = virt4, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vds='Host[virt4,5ae29ed2-9c0f-4754-8df3-b43d14726846]'})' execution failed: Recovering from crash or Initializing 2017-10-01 15:24:59,581-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [224c57f9] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command GetCapabilitiesVDS failed: Recovering from crash or Initializing 2017-10-01 15:24:59,581-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler4) [224c57f9] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@7465c8d0' 2017-10-01 15:24:59,581-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler4) [224c57f9] HostName = virt4 2017-10-01 15:24:59,581-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler4) [224c57f9] Command 'GetCapabilitiesVDSCommand(HostName = virt4, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vds='Host[virt4,5ae29ed2-9c0f-4754-8df3-b43d14726846]'})' execution failed: Recovering from crash or Initializing 2017-10-01 15:25:00,584-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [5532121c] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt1 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:25:00,584-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [5532121c] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 15:25:00,584-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [5532121c] HostName = virt1 2017-10-01 15:25:00,584-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [5532121c] Command 'ConnectStoragePoolVDSCommand(HostName = virt1, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vdsId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:25:00,584-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [5532121c] FINISH, ConnectStoragePoolVDSCommand, log id: 5605c5bd 2017-10-01 15:25:00,584-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler5) [5532121c] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:25:02,556-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler8) [4474e308] Command 'GetAllVmStatsVDSCommand(HostName = virt4, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846'})' execution failed: Recovering from crash or Initializing 2017-10-01 15:25:02,556-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.PollVmStatsRefresher] (DefaultQuartzScheduler8) [4474e308] Failed to fetch vms info for host 'virt4' - skipping VMs monitoring. 2017-10-01 15:25:03,603-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [15bc7539] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command GetCapabilitiesVDS failed: Recovering from crash or Initializing 2017-10-01 15:25:03,603-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler9) [15bc7539] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@151d5542' 2017-10-01 15:25:03,604-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler9) [15bc7539] HostName = virt4 2017-10-01 15:25:03,604-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler9) [15bc7539] Command 'GetCapabilitiesVDSCommand(HostName = virt4, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vds='Host[virt4,5ae29ed2-9c0f-4754-8df3-b43d14726846]'})' execution failed: Recovering from crash or Initializing 2017-10-01 15:25:06,663-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [507eb2eb] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command GetCapabilitiesVDS failed: Recovering from crash or Initializing 2017-10-01 15:25:06,663-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler10) [507eb2eb] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@60c15ff5' 2017-10-01 15:25:06,663-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler10) [507eb2eb] HostName = virt4 2017-10-01 15:25:06,663-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler10) [507eb2eb] Command 'GetCapabilitiesVDSCommand(HostName = virt4, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vds='Host[virt4,5ae29ed2-9c0f-4754-8df3-b43d14726846]'})' execution failed: Recovering from crash or Initializing 2017-10-01 15:25:09,729-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2481c3b0] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command GetCapabilitiesVDS failed: Recovering from crash or Initializing 2017-10-01 15:25:09,729-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler8) [2481c3b0] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@1ac9c40a' 2017-10-01 15:25:09,729-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler8) [2481c3b0] HostName = virt4 2017-10-01 15:25:09,729-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler8) [2481c3b0] Command 'GetCapabilitiesVDSCommand(HostName = virt4, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vds='Host[virt4,5ae29ed2-9c0f-4754-8df3-b43d14726846]'})' execution failed: Recovering from crash or Initializing 2017-10-01 15:25:13,750-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [31bb627f] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command GetCapabilitiesVDS failed: Recovering from crash or Initializing 2017-10-01 15:25:13,750-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler2) [31bb627f] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@1b657b81' 2017-10-01 15:25:13,750-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler2) [31bb627f] HostName = virt4 2017-10-01 15:25:13,750-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler2) [31bb627f] Command 'GetCapabilitiesVDSCommand(HostName = virt4, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vds='Host[virt4,5ae29ed2-9c0f-4754-8df3-b43d14726846]'})' execution failed: Recovering from crash or Initializing 2017-10-01 15:25:17,561-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [797295d] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command GetCapabilitiesVDS failed: Recovering from crash or Initializing 2017-10-01 15:25:17,561-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler4) [797295d] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@9da2e1b' 2017-10-01 15:25:17,561-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler4) [797295d] HostName = virt4 2017-10-01 15:25:17,561-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler4) [797295d] Command 'GetCapabilitiesVDSCommand(HostName = virt4, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vds='Host[virt4,5ae29ed2-9c0f-4754-8df3-b43d14726846]'})' execution failed: Recovering from crash or Initializing 2017-10-01 15:25:17,607-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler7) [1ecf79e8] Command 'GetAllVmStatsVDSCommand(HostName = virt4, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846'})' execution failed: Recovering from crash or Initializing 2017-10-01 15:25:17,607-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.PollVmStatsRefresher] (DefaultQuartzScheduler7) [1ecf79e8] Failed to fetch vms info for host 'virt4' - skipping VMs monitoring. 2017-10-01 15:25:21,583-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [4ff9a775] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command GetCapabilitiesVDS failed: Recovering from crash or Initializing 2017-10-01 15:25:21,583-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler4) [4ff9a775] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@1434f786' 2017-10-01 15:25:21,583-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler4) [4ff9a775] HostName = virt4 2017-10-01 15:25:21,583-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler4) [4ff9a775] Command 'GetCapabilitiesVDSCommand(HostName = virt4, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vds='Host[virt4,5ae29ed2-9c0f-4754-8df3-b43d14726846]'})' execution failed: Recovering from crash or Initializing 2017-10-01 15:25:24,759-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [81a015c] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command GetCapabilitiesVDS failed: Recovering from crash or Initializing 2017-10-01 15:25:24,759-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler3) [81a015c] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@23ad3fad' 2017-10-01 15:25:24,760-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler3) [81a015c] HostName = virt4 2017-10-01 15:25:24,760-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler3) [81a015c] Command 'GetCapabilitiesVDSCommand(HostName = virt4, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vds='Host[virt4,5ae29ed2-9c0f-4754-8df3-b43d14726846]'})' execution failed: Recovering from crash or Initializing 2017-10-01 15:25:28,782-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [79b0d27] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command GetCapabilitiesVDS failed: Recovering from crash or Initializing 2017-10-01 15:25:28,782-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler6) [79b0d27] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@196d5dfe' 2017-10-01 15:25:28,782-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler6) [79b0d27] HostName = virt4 2017-10-01 15:25:28,782-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler6) [79b0d27] Command 'GetCapabilitiesVDSCommand(HostName = virt4, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vds='Host[virt4,5ae29ed2-9c0f-4754-8df3-b43d14726846]'})' execution failed: Recovering from crash or Initializing 2017-10-01 15:25:32,614-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [467a0f4b] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command GetCapabilitiesVDS failed: Recovering from crash or Initializing 2017-10-01 15:25:32,614-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler10) [467a0f4b] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@2f88605e' 2017-10-01 15:25:32,614-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler10) [467a0f4b] HostName = virt4 2017-10-01 15:25:32,614-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler10) [467a0f4b] Command 'GetCapabilitiesVDSCommand(HostName = virt4, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vds='Host[virt4,5ae29ed2-9c0f-4754-8df3-b43d14726846]'})' execution failed: Recovering from crash or Initializing 2017-10-01 15:25:33,611-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler7) [67a74686] Command 'GetAllVmStatsVDSCommand(HostName = virt4, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846'})' execution failed: Recovering from crash or Initializing 2017-10-01 15:25:33,611-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.PollVmStatsRefresher] (DefaultQuartzScheduler7) [67a74686] Failed to fetch vms info for host 'virt4' - skipping VMs monitoring. 2017-10-01 15:25:36,638-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [7cf19a78] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command GetCapabilitiesVDS failed: Recovering from crash or Initializing 2017-10-01 15:25:36,638-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler1) [7cf19a78] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@500fdfb5' 2017-10-01 15:25:36,638-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler1) [7cf19a78] HostName = virt4 2017-10-01 15:25:36,638-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler1) [7cf19a78] Command 'GetCapabilitiesVDSCommand(HostName = virt4, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vds='Host[virt4,5ae29ed2-9c0f-4754-8df3-b43d14726846]'})' execution failed: Recovering from crash or Initializing 2017-10-01 15:25:40,660-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [43903e5e] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command GetCapabilitiesVDS failed: Recovering from crash or Initializing 2017-10-01 15:25:40,660-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler1) [43903e5e] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@5ceaf6e9' 2017-10-01 15:25:40,660-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler1) [43903e5e] HostName = virt4 2017-10-01 15:25:40,660-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler1) [43903e5e] Command 'GetCapabilitiesVDSCommand(HostName = virt4, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vds='Host[virt4,5ae29ed2-9c0f-4754-8df3-b43d14726846]'})' execution failed: Recovering from crash or Initializing 2017-10-01 15:25:44,683-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [3721e6ac] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command GetCapabilitiesVDS failed: Recovering from crash or Initializing 2017-10-01 15:25:44,683-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler7) [3721e6ac] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@7d7a1f9c' 2017-10-01 15:25:44,683-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler7) [3721e6ac] HostName = virt4 2017-10-01 15:25:44,683-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler7) [3721e6ac] Command 'GetCapabilitiesVDSCommand(HostName = virt4, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vds='Host[virt4,5ae29ed2-9c0f-4754-8df3-b43d14726846]'})' execution failed: Recovering from crash or Initializing 2017-10-01 15:25:48,617-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [1c49b681] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command GetCapabilitiesVDS failed: Recovering from crash or Initializing 2017-10-01 15:25:48,617-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler8) [1c49b681] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@4436743' 2017-10-01 15:25:48,617-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler8) [1c49b681] HostName = virt4 2017-10-01 15:25:48,617-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler8) [1c49b681] Command 'GetCapabilitiesVDSCommand(HostName = virt4, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vds='Host[virt4,5ae29ed2-9c0f-4754-8df3-b43d14726846]'})' execution failed: Recovering from crash or Initializing 2017-10-01 15:25:49,615-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler4) [8f900b7] Command 'GetAllVmStatsVDSCommand(HostName = virt4, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846'})' execution failed: Recovering from crash or Initializing 2017-10-01 15:25:49,615-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.PollVmStatsRefresher] (DefaultQuartzScheduler4) [8f900b7] Failed to fetch vms info for host 'virt4' - skipping VMs monitoring. 2017-10-01 15:25:52,639-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [3efedb89] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command GetCapabilitiesVDS failed: Recovering from crash or Initializing 2017-10-01 15:25:52,639-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler7) [3efedb89] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@42eff9aa' 2017-10-01 15:25:52,639-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler7) [3efedb89] HostName = virt4 2017-10-01 15:25:52,639-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler7) [3efedb89] Command 'GetCapabilitiesVDSCommand(HostName = virt4, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vds='Host[virt4,5ae29ed2-9c0f-4754-8df3-b43d14726846]'})' execution failed: Recovering from crash or Initializing 2017-10-01 15:25:55,502-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [582b571f] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 15:25:55,779-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [56290a2e] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command GetCapabilitiesVDS failed: Recovering from crash or Initializing 2017-10-01 15:25:55,779-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler1) [56290a2e] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@435651bc' 2017-10-01 15:25:55,779-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler1) [56290a2e] HostName = virt4 2017-10-01 15:25:55,779-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler1) [56290a2e] Command 'GetCapabilitiesVDSCommand(HostName = virt4, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vds='Host[virt4,5ae29ed2-9c0f-4754-8df3-b43d14726846]'})' execution failed: Recovering from crash or Initializing 2017-10-01 15:25:59,801-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [4cb911ed] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command GetCapabilitiesVDS failed: Recovering from crash or Initializing 2017-10-01 15:25:59,801-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler1) [4cb911ed] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@605d21e7' 2017-10-01 15:25:59,801-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler1) [4cb911ed] HostName = virt4 2017-10-01 15:25:59,801-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler1) [4cb911ed] Command 'GetCapabilitiesVDSCommand(HostName = virt4, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vds='Host[virt4,5ae29ed2-9c0f-4754-8df3-b43d14726846]'})' execution failed: Recovering from crash or Initializing 2017-10-01 15:26:03,822-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [39f3b7b8] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command GetCapabilitiesVDS failed: Recovering from crash or Initializing 2017-10-01 15:26:03,822-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler2) [39f3b7b8] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@4a15c77' 2017-10-01 15:26:03,822-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler2) [39f3b7b8] HostName = virt4 2017-10-01 15:26:03,822-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler2) [39f3b7b8] Command 'GetCapabilitiesVDSCommand(HostName = virt4, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vds='Host[virt4,5ae29ed2-9c0f-4754-8df3-b43d14726846]'})' execution failed: Recovering from crash or Initializing 2017-10-01 15:26:05,618-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler3) [8edf8fc] Command 'GetAllVmStatsVDSCommand(HostName = virt4, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846'})' execution failed: Recovering from crash or Initializing 2017-10-01 15:26:05,618-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.PollVmStatsRefresher] (DefaultQuartzScheduler3) [8edf8fc] Failed to fetch vms info for host 'virt4' - skipping VMs monitoring. 2017-10-01 15:26:07,845-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [63569193] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command GetCapabilitiesVDS failed: Recovering from crash or Initializing 2017-10-01 15:26:07,845-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler2) [63569193] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@5d1df1da' 2017-10-01 15:26:07,845-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler2) [63569193] HostName = virt4 2017-10-01 15:26:07,845-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler2) [63569193] Command 'GetCapabilitiesVDSCommand(HostName = virt4, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vds='Host[virt4,5ae29ed2-9c0f-4754-8df3-b43d14726846]'})' execution failed: Recovering from crash or Initializing 2017-10-01 15:26:11,099-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [3813f7f2] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command GetCapabilitiesVDS failed: Recovering from crash or Initializing 2017-10-01 15:26:11,099-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler4) [3813f7f2] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@5757429a' 2017-10-01 15:26:11,099-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler4) [3813f7f2] HostName = virt4 2017-10-01 15:26:11,099-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler4) [3813f7f2] Command 'GetCapabilitiesVDSCommand(HostName = virt4, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vds='Host[virt4,5ae29ed2-9c0f-4754-8df3-b43d14726846]'})' execution failed: Recovering from crash or Initializing 2017-10-01 15:26:15,121-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [43f0ab2f] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command GetCapabilitiesVDS failed: Recovering from crash or Initializing 2017-10-01 15:26:15,121-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler9) [43f0ab2f] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@45950d2' 2017-10-01 15:26:15,121-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler9) [43f0ab2f] HostName = virt4 2017-10-01 15:26:15,121-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler9) [43f0ab2f] Command 'GetCapabilitiesVDSCommand(HostName = virt4, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vds='Host[virt4,5ae29ed2-9c0f-4754-8df3-b43d14726846]'})' execution failed: Recovering from crash or Initializing 2017-10-01 15:26:19,143-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [57bf6a8b] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command GetCapabilitiesVDS failed: Recovering from crash or Initializing 2017-10-01 15:26:19,143-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler2) [57bf6a8b] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@23db8e4c' 2017-10-01 15:26:19,143-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler2) [57bf6a8b] HostName = virt4 2017-10-01 15:26:19,143-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler2) [57bf6a8b] Command 'GetCapabilitiesVDSCommand(HostName = virt4, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vds='Host[virt4,5ae29ed2-9c0f-4754-8df3-b43d14726846]'})' execution failed: Recovering from crash or Initializing 2017-10-01 15:26:21,622-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler2) [522d4b9a] Command 'GetAllVmStatsVDSCommand(HostName = virt4, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846'})' execution failed: Recovering from crash or Initializing 2017-10-01 15:26:21,622-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.PollVmStatsRefresher] (DefaultQuartzScheduler2) [522d4b9a] Failed to fetch vms info for host 'virt4' - skipping VMs monitoring. 2017-10-01 15:26:23,166-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [29427087] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command GetCapabilitiesVDS failed: Recovering from crash or Initializing 2017-10-01 15:26:23,166-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler1) [29427087] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@2d5880d1' 2017-10-01 15:26:23,166-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler1) [29427087] HostName = virt4 2017-10-01 15:26:23,166-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler1) [29427087] Command 'GetCapabilitiesVDSCommand(HostName = virt4, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vds='Host[virt4,5ae29ed2-9c0f-4754-8df3-b43d14726846]'})' execution failed: Recovering from crash or Initializing 2017-10-01 15:26:26,207-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [3f99cfe3] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command GetCapabilitiesVDS failed: Recovering from crash or Initializing 2017-10-01 15:26:26,207-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler1) [3f99cfe3] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@721fb251' 2017-10-01 15:26:26,207-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler1) [3f99cfe3] HostName = virt4 2017-10-01 15:26:26,207-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler1) [3f99cfe3] Command 'GetCapabilitiesVDSCommand(HostName = virt4, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vds='Host[virt4,5ae29ed2-9c0f-4754-8df3-b43d14726846]'})' execution failed: Recovering from crash or Initializing 2017-10-01 15:26:30,228-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [19d482c] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command GetCapabilitiesVDS failed: Recovering from crash or Initializing 2017-10-01 15:26:30,229-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler6) [19d482c] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@708efecd' 2017-10-01 15:26:30,229-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler6) [19d482c] HostName = virt4 2017-10-01 15:26:30,229-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler6) [19d482c] Command 'GetCapabilitiesVDSCommand(HostName = virt4, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vds='Host[virt4,5ae29ed2-9c0f-4754-8df3-b43d14726846]'})' execution failed: Recovering from crash or Initializing 2017-10-01 15:26:34,252-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [28d829cb] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command GetCapabilitiesVDS failed: Recovering from crash or Initializing 2017-10-01 15:26:34,252-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler9) [28d829cb] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@7a7ccfb7' 2017-10-01 15:26:34,252-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler9) [28d829cb] HostName = virt4 2017-10-01 15:26:34,252-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler9) [28d829cb] Command 'GetCapabilitiesVDSCommand(HostName = virt4, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vds='Host[virt4,5ae29ed2-9c0f-4754-8df3-b43d14726846]'})' execution failed: Recovering from crash or Initializing 2017-10-01 15:26:37,169-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler10) [1715b76d] Command 'GetAllVmStatsVDSCommand(HostName = virt4, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846'})' execution failed: Recovering from crash or Initializing 2017-10-01 15:26:37,169-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.PollVmStatsRefresher] (DefaultQuartzScheduler10) [1715b76d] Failed to fetch vms info for host 'virt4' - skipping VMs monitoring. 2017-10-01 15:26:38,274-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [3015a9e9] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command GetCapabilitiesVDS failed: Recovering from crash or Initializing 2017-10-01 15:26:38,274-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler6) [3015a9e9] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@72ba6f5c' 2017-10-01 15:26:38,274-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler6) [3015a9e9] HostName = virt4 2017-10-01 15:26:38,274-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler6) [3015a9e9] Command 'GetCapabilitiesVDSCommand(HostName = virt4, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vds='Host[virt4,5ae29ed2-9c0f-4754-8df3-b43d14726846]'})' execution failed: Recovering from crash or Initializing 2017-10-01 15:26:42,296-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [2bfae31b] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command GetCapabilitiesVDS failed: Recovering from crash or Initializing 2017-10-01 15:26:42,296-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler2) [2bfae31b] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@540bb827' 2017-10-01 15:26:42,296-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler2) [2bfae31b] HostName = virt4 2017-10-01 15:26:42,296-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler2) [2bfae31b] Command 'GetCapabilitiesVDSCommand(HostName = virt4, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vds='Host[virt4,5ae29ed2-9c0f-4754-8df3-b43d14726846]'})' execution failed: Recovering from crash or Initializing 2017-10-01 15:26:46,318-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [68ed3527] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command GetCapabilitiesVDS failed: Recovering from crash or Initializing 2017-10-01 15:26:46,318-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler7) [68ed3527] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@3b233755' 2017-10-01 15:26:46,318-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler7) [68ed3527] HostName = virt4 2017-10-01 15:26:46,318-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler7) [68ed3527] Command 'GetCapabilitiesVDSCommand(HostName = virt4, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vds='Host[virt4,5ae29ed2-9c0f-4754-8df3-b43d14726846]'})' execution failed: Recovering from crash or Initializing 2017-10-01 15:26:50,340-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [2998996b] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command GetCapabilitiesVDS failed: Recovering from crash or Initializing 2017-10-01 15:26:50,340-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler10) [2998996b] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@11c9c6e3' 2017-10-01 15:26:50,340-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler10) [2998996b] HostName = virt4 2017-10-01 15:26:50,340-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler10) [2998996b] Command 'GetCapabilitiesVDSCommand(HostName = virt4, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vds='Host[virt4,5ae29ed2-9c0f-4754-8df3-b43d14726846]'})' execution failed: Recovering from crash or Initializing 2017-10-01 15:26:52,170-04 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to virt4-mgmt/192.168.0.37 2017-10-01 15:26:52,280-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler3) [575b7dac] Command 'GetAllVmStatsVDSCommand(HostName = virt4, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846'})' execution failed: Recovering from crash or Initializing 2017-10-01 15:26:52,280-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.PollVmStatsRefresher] (DefaultQuartzScheduler3) [575b7dac] Failed to fetch vms info for host 'virt4' - skipping VMs monitoring. 2017-10-01 15:26:54,362-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [5a54604a] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command GetCapabilitiesVDS failed: Recovering from crash or Initializing 2017-10-01 15:26:54,362-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler10) [5a54604a] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@1d4d4a8f' 2017-10-01 15:26:54,362-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler10) [5a54604a] HostName = virt4 2017-10-01 15:26:54,362-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler10) [5a54604a] Command 'GetCapabilitiesVDSCommand(HostName = virt4, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vds='Host[virt4,5ae29ed2-9c0f-4754-8df3-b43d14726846]'})' execution failed: Recovering from crash or Initializing 2017-10-01 15:26:55,520-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [194749b6] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 15:26:58,406-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [1fb081de] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command GetCapabilitiesVDS failed: Recovering from crash or Initializing 2017-10-01 15:26:58,406-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler10) [1fb081de] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@416d6577' 2017-10-01 15:26:58,406-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler10) [1fb081de] HostName = virt4 2017-10-01 15:26:58,406-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler10) [1fb081de] Command 'GetCapabilitiesVDSCommand(HostName = virt4, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vds='Host[virt4,5ae29ed2-9c0f-4754-8df3-b43d14726846]'})' execution failed: Recovering from crash or Initializing 2017-10-01 15:27:00,590-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [5532121c] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt1 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 15:27:00,590-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler5) [5532121c] Command 'SpmStatusVDSCommand(HostName = virt1, SpmStatusVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 15:27:00,590-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler5) [5532121c] Irs placed on server '45f8b331-842e-48e7-9df8-56adddb93836' failed. Proceed Failover 2017-10-01 15:27:00,648-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [5532121c] START, ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 18e0ca61 2017-10-01 15:27:00,648-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [5532121c] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 15:27:02,428-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [3ada7239] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command GetCapabilitiesVDS failed: Recovering from crash or Initializing 2017-10-01 15:27:02,428-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler4) [3ada7239] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@37ac6520' 2017-10-01 15:27:02,428-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler4) [3ada7239] HostName = virt4 2017-10-01 15:27:02,428-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler4) [3ada7239] Command 'GetCapabilitiesVDSCommand(HostName = virt4, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vds='Host[virt4,5ae29ed2-9c0f-4754-8df3-b43d14726846]'})' execution failed: Recovering from crash or Initializing 2017-10-01 15:27:06,450-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [793ce134] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command GetCapabilitiesVDS failed: Recovering from crash or Initializing 2017-10-01 15:27:06,451-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler9) [793ce134] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@793ae720' 2017-10-01 15:27:06,451-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler9) [793ce134] HostName = virt4 2017-10-01 15:27:06,451-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler9) [793ce134] Command 'GetCapabilitiesVDSCommand(HostName = virt4, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vds='Host[virt4,5ae29ed2-9c0f-4754-8df3-b43d14726846]'})' execution failed: Recovering from crash or Initializing 2017-10-01 15:27:08,283-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler1) [19b38243] Command 'GetAllVmStatsVDSCommand(HostName = virt4, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846'})' execution failed: Recovering from crash or Initializing 2017-10-01 15:27:08,283-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.PollVmStatsRefresher] (DefaultQuartzScheduler1) [19b38243] Failed to fetch vms info for host 'virt4' - skipping VMs monitoring. 2017-10-01 15:27:10,472-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [3a87535e] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command GetCapabilitiesVDS failed: Recovering from crash or Initializing 2017-10-01 15:27:10,473-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler9) [3a87535e] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@7adac3a3' 2017-10-01 15:27:10,473-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler9) [3a87535e] HostName = virt4 2017-10-01 15:27:10,473-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler9) [3a87535e] Command 'GetCapabilitiesVDSCommand(HostName = virt4, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vds='Host[virt4,5ae29ed2-9c0f-4754-8df3-b43d14726846]'})' execution failed: Recovering from crash or Initializing 2017-10-01 15:27:14,497-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [4da81473] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command GetCapabilitiesVDS failed: Recovering from crash or Initializing 2017-10-01 15:27:14,497-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler7) [4da81473] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@25a72bc2' 2017-10-01 15:27:14,497-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler7) [4da81473] HostName = virt4 2017-10-01 15:27:14,497-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler7) [4da81473] Command 'GetCapabilitiesVDSCommand(HostName = virt4, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vds='Host[virt4,5ae29ed2-9c0f-4754-8df3-b43d14726846]'})' execution failed: Recovering from crash or Initializing 2017-10-01 15:27:18,520-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [61a25d22] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command GetCapabilitiesVDS failed: Recovering from crash or Initializing 2017-10-01 15:27:18,520-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler1) [61a25d22] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@550092fb' 2017-10-01 15:27:18,520-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler1) [61a25d22] HostName = virt4 2017-10-01 15:27:18,520-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler1) [61a25d22] Command 'GetCapabilitiesVDSCommand(HostName = virt4, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vds='Host[virt4,5ae29ed2-9c0f-4754-8df3-b43d14726846]'})' execution failed: Recovering from crash or Initializing 2017-10-01 15:27:22,542-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [fc7fc9f] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command GetCapabilitiesVDS failed: Recovering from crash or Initializing 2017-10-01 15:27:22,542-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler9) [fc7fc9f] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@74551861' 2017-10-01 15:27:22,542-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler9) [fc7fc9f] HostName = virt4 2017-10-01 15:27:22,542-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler9) [fc7fc9f] Command 'GetCapabilitiesVDSCommand(HostName = virt4, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vds='Host[virt4,5ae29ed2-9c0f-4754-8df3-b43d14726846]'})' execution failed: Recovering from crash or Initializing 2017-10-01 15:27:24,288-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler3) [2902b163] Command 'GetAllVmStatsVDSCommand(HostName = virt4, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846'})' execution failed: Recovering from crash or Initializing 2017-10-01 15:27:24,288-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.PollVmStatsRefresher] (DefaultQuartzScheduler3) [2902b163] Failed to fetch vms info for host 'virt4' - skipping VMs monitoring. 2017-10-01 15:27:25,937-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [2060cc09] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command GetCapabilitiesVDS failed: Recovering from crash or Initializing 2017-10-01 15:27:25,937-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler10) [2060cc09] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@771b38aa' 2017-10-01 15:27:25,937-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler10) [2060cc09] HostName = virt4 2017-10-01 15:27:25,937-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler10) [2060cc09] Command 'GetCapabilitiesVDSCommand(HostName = virt4, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vds='Host[virt4,5ae29ed2-9c0f-4754-8df3-b43d14726846]'})' execution failed: Recovering from crash or Initializing 2017-10-01 15:27:29,762-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [6280b728] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command GetCapabilitiesVDS failed: Recovering from crash or Initializing 2017-10-01 15:27:29,762-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler10) [6280b728] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@23b0c2d9' 2017-10-01 15:27:29,762-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler10) [6280b728] HostName = virt4 2017-10-01 15:27:29,762-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler10) [6280b728] Command 'GetCapabilitiesVDSCommand(HostName = virt4, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vds='Host[virt4,5ae29ed2-9c0f-4754-8df3-b43d14726846]'})' execution failed: Recovering from crash or Initializing 2017-10-01 15:27:33,785-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3ddaaf37] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command GetCapabilitiesVDS failed: Recovering from crash or Initializing 2017-10-01 15:27:33,785-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler2) [3ddaaf37] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@6fceb37c' 2017-10-01 15:27:33,785-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler2) [3ddaaf37] HostName = virt4 2017-10-01 15:27:33,785-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler2) [3ddaaf37] Command 'GetCapabilitiesVDSCommand(HostName = virt4, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vds='Host[virt4,5ae29ed2-9c0f-4754-8df3-b43d14726846]'})' execution failed: Recovering from crash or Initializing 2017-10-01 15:27:36,529-04 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler9) [27ccae48] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5ae29ed2-9c0f-4754-8df3-b43d14726846 Type: VDS 2017-10-01 15:27:36,544-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [27ccae48] START, SetVdsStatusVDSCommand(HostName = virt4, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', status='NonOperational', nonOperationalReason='TIMEOUT_RECOVERING_FROM_CRASH', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 11883581 2017-10-01 15:27:36,544-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [27ccae48] VDS 'virt4' is spm and moved from up calling resetIrs. 2017-10-01 15:27:36,544-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (DefaultQuartzScheduler9) [27ccae48] START, ResetIrsVDSCommand( ResetIrsVDSCommandParameters:{runAsync='true', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', ignoreFailoverLimit='false', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', ignoreStopFailed='false'}), log id: 58d07231 2017-10-01 15:27:36,558-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (DefaultQuartzScheduler9) [27ccae48] START, SpmStopVDSCommand(HostName = virt4, SpmStopVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'}), log id: 7cc9139b 2017-10-01 15:27:37,567-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [27ccae48] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command HSMGetAllTasksStatusesVDS failed: Recovering from crash or Initializing 2017-10-01 15:27:37,567-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler9) [27ccae48] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand' return value ' 2017-10-01 15:27:37,568-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler9) [27ccae48] HostName = virt4 2017-10-01 15:27:37,568-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler9) [27ccae48] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = virt4, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846'})' execution failed: Recovering from crash or Initializing 2017-10-01 15:27:37,568-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (DefaultQuartzScheduler9) [27ccae48] SpmStopVDSCommand::Stopping SPM on vds 'virt4', pool id 'aca61eac-ed8a-484b-bbee-3645148f6405' 2017-10-01 15:27:38,575-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [27ccae48] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command SpmStopVDS failed: Recovering from crash or Initializing 2017-10-01 15:27:38,576-04 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (DefaultQuartzScheduler9) [27ccae48] Could not stop spm of pool 'aca61eac-ed8a-484b-bbee-3645148f6405' on vds '5ae29ed2-9c0f-4754-8df3-b43d14726846': Recovering from crash or Initializing 2017-10-01 15:27:38,576-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (DefaultQuartzScheduler9) [27ccae48] FINISH, SpmStopVDSCommand, log id: 7cc9139b 2017-10-01 15:27:38,576-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (DefaultQuartzScheduler9) [27ccae48] FINISH, ResetIrsVDSCommand, log id: 58d07231 2017-10-01 15:27:38,577-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler9) [27ccae48] FINISH, SetVdsStatusVDSCommand, log id: 11883581 2017-10-01 15:27:38,597-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [27ccae48] EVENT_ID: VDS_RECOVER_FAILED(19), Correlation ID: 27ccae48, Job ID: f50f11fd-a14a-4072-9a32-a32b0742f349, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host virt4 failed to recover. 2017-10-01 15:27:40,291-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler6) [559c62d] Command 'GetAllVmStatsVDSCommand(HostName = virt4, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846'})' execution failed: Recovering from crash or Initializing 2017-10-01 15:27:40,291-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.PollVmStatsRefresher] (DefaultQuartzScheduler6) [559c62d] Failed to fetch vms info for host 'virt4' - skipping VMs monitoring. 2017-10-01 15:27:45,647-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [1e625093] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command GetStatsVDS failed: Recovering from crash or Initializing 2017-10-01 15:27:45,647-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStatsVDSCommand] (DefaultQuartzScheduler7) [1e625093] Command 'GetStatsVDSCommand(HostName = virt4, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vds='Host[virt4,5ae29ed2-9c0f-4754-8df3-b43d14726846]'})' execution failed: Recovering from crash or Initializing 2017-10-01 15:27:45,647-04 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [1e625093] Failed getting vds stats, host='virt4'(5ae29ed2-9c0f-4754-8df3-b43d14726846): org.ovirt.engine.core.vdsbroker.vdsbroker.VDSRecoveringException: Recovering from crash or Initializing 2017-10-01 15:27:55,543-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [75939bd0] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 15:27:55,968-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler6) [36d2bd5d] Command 'GetAllVmStatsVDSCommand(HostName = virt4, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846'})' execution failed: Recovering from crash or Initializing 2017-10-01 15:27:55,968-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.PollVmStatsRefresher] (DefaultQuartzScheduler6) [36d2bd5d] Failed to fetch vms info for host 'virt4' - skipping VMs monitoring. 2017-10-01 15:28:01,730-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [7ea70cd7] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command GetStatsVDS failed: Recovering from crash or Initializing 2017-10-01 15:28:01,730-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStatsVDSCommand] (DefaultQuartzScheduler1) [7ea70cd7] Command 'GetStatsVDSCommand(HostName = virt4, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vds='Host[virt4,5ae29ed2-9c0f-4754-8df3-b43d14726846]'})' execution failed: Recovering from crash or Initializing 2017-10-01 15:28:01,730-04 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [7ea70cd7] Failed getting vds stats, host='virt4'(5ae29ed2-9c0f-4754-8df3-b43d14726846): org.ovirt.engine.core.vdsbroker.vdsbroker.VDSRecoveringException: Recovering from crash or Initializing 2017-10-01 15:28:04,463-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-7) [] starting processDomainRecovery for domain 'c8653dba-63ec-4447-872c-0ffcc6af9c0f:data'. 2017-10-01 15:28:04,526-04 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-7) [] Domain 'c8653dba-63ec-4447-872c-0ffcc6af9c0f:data' was reported by all hosts in status UP as problematic. Not moving the domain to NonOperational because it is being reconstructed now. 2017-10-01 15:28:04,603-04 INFO [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-7) [737626d0] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: c8653dba-63ec-4447-872c-0ffcc6af9c0f Type: Storage 2017-10-01 15:28:04,695-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-6-thread-7) [737626d0] START, ResetIrsVDSCommand( ResetIrsVDSCommandParameters:{runAsync='true', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', ignoreFailoverLimit='false', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', ignoreStopFailed='true'}), log id: 50213ca4 2017-10-01 15:28:04,711-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-6-thread-7) [737626d0] START, SpmStopVDSCommand(HostName = virt4, SpmStopVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'}), log id: 1674833e 2017-10-01 15:28:05,721-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-7) [737626d0] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command HSMGetAllTasksStatusesVDS failed: Recovering from crash or Initializing 2017-10-01 15:28:05,721-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (org.ovirt.thread.pool-6-thread-7) [737626d0] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand' return value ' 2017-10-01 15:28:05,721-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (org.ovirt.thread.pool-6-thread-7) [737626d0] HostName = virt4 2017-10-01 15:28:05,721-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (org.ovirt.thread.pool-6-thread-7) [737626d0] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = virt4, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846'})' execution failed: Recovering from crash or Initializing 2017-10-01 15:28:05,721-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-6-thread-7) [737626d0] SpmStopVDSCommand::Stopping SPM on vds 'virt4', pool id 'aca61eac-ed8a-484b-bbee-3645148f6405' 2017-10-01 15:28:06,729-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-7) [737626d0] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command SpmStopVDS failed: Recovering from crash or Initializing 2017-10-01 15:28:06,729-04 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-6-thread-7) [737626d0] Could not stop spm of pool 'aca61eac-ed8a-484b-bbee-3645148f6405' on vds '5ae29ed2-9c0f-4754-8df3-b43d14726846': Recovering from crash or Initializing 2017-10-01 15:28:06,729-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-6-thread-7) [737626d0] FINISH, SpmStopVDSCommand, log id: 1674833e 2017-10-01 15:28:06,731-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-6-thread-7) [737626d0] FINISH, ResetIrsVDSCommand, log id: 50213ca4 2017-10-01 15:28:06,731-04 WARN [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-7) [737626d0] skipping connect and refresh for all hosts, last master 'true', command status 'true' 2017-10-01 15:28:06,747-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-7) [737626d0] EVENT_ID: RECONSTRUCT_MASTER_FAILED(985), Correlation ID: 737626d0, Job ID: 391e885b-4f98-4f5e-966f-dda2108723fd, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center EECS-Primary. 2017-10-01 15:28:06,750-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-7) [737626d0] Removing vds '[]' from the domain in maintenance cache 2017-10-01 15:28:06,750-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-7) [737626d0] Removing host(s) '[]' from hosts unseen domain report cache 2017-10-01 15:28:06,750-04 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-6-thread-44) [15de2e31] Finished reconstruct for pool 'aca61eac-ed8a-484b-bbee-3645148f6405'. Clearing event queue 2017-10-01 15:28:06,752-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-32) [] starting processDomainRecovery for domain 'f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2:export'. 2017-10-01 15:28:06,813-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-32) [] Domain 'f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2:export' was reported by all hosts in status UP as problematic. Moving the domain to NonOperational. 2017-10-01 15:28:06,829-04 INFO [org.ovirt.engine.core.bll.storage.domain.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-6-thread-32) [5aaeee04] Lock Acquired to object 'EngineLock:{exclusiveLocks='[f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=STORAGE]', sharedLocks=''}' 2017-10-01 15:28:06,834-04 INFO [org.ovirt.engine.core.bll.storage.domain.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-6-thread-32) [5aaeee04] Running command: DeactivateStorageDomainCommand internal: true. Entities affected : ID: f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2 Type: StorageAction group MANIPULATE_STORAGE_DOMAIN with role type ADMIN 2017-10-01 15:28:06,836-04 INFO [org.ovirt.engine.core.bll.storage.domain.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-6-thread-32) [5aaeee04] Lock freed to object 'EngineLock:{exclusiveLocks='[f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=STORAGE]', sharedLocks=''}' 2017-10-01 15:28:06,842-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-32) [5aaeee04] EVENT_ID: SYSTEM_DEACTIVATED_STORAGE_DOMAIN(970), Correlation ID: 5aaeee04, Job ID: d6097d0a-f2d7-41b2-b2cc-3af498423e38, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Storage Domain export (Data Center EECS-Primary) was deactivated by system because it's not visible by any of the hosts. 2017-10-01 15:28:06,845-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-32) [5aaeee04] Removing vds '[]' from the domain in maintenance cache 2017-10-01 15:28:06,845-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-32) [5aaeee04] Removing host(s) '[]' from hosts unseen domain report cache 2017-10-01 15:28:06,845-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-2) [] starting processDomainRecovery for domain '63d680c7-4c94-4b29-a6cb-f288a128b9bc:iso'. 2017-10-01 15:28:06,900-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-2) [] Domain '63d680c7-4c94-4b29-a6cb-f288a128b9bc:iso' was reported by all hosts in status UP as problematic. Moving the domain to NonOperational. 2017-10-01 15:28:06,911-04 INFO [org.ovirt.engine.core.bll.storage.domain.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-6-thread-2) [e5acb36] Lock Acquired to object 'EngineLock:{exclusiveLocks='[63d680c7-4c94-4b29-a6cb-f288a128b9bc=STORAGE]', sharedLocks=''}' 2017-10-01 15:28:06,917-04 INFO [org.ovirt.engine.core.bll.storage.domain.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-6-thread-2) [e5acb36] Running command: DeactivateStorageDomainCommand internal: true. Entities affected : ID: 63d680c7-4c94-4b29-a6cb-f288a128b9bc Type: StorageAction group MANIPULATE_STORAGE_DOMAIN with role type ADMIN 2017-10-01 15:28:06,919-04 INFO [org.ovirt.engine.core.bll.storage.domain.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-6-thread-2) [e5acb36] Lock freed to object 'EngineLock:{exclusiveLocks='[63d680c7-4c94-4b29-a6cb-f288a128b9bc=STORAGE]', sharedLocks=''}' 2017-10-01 15:28:06,924-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-2) [e5acb36] EVENT_ID: SYSTEM_DEACTIVATED_STORAGE_DOMAIN(970), Correlation ID: e5acb36, Job ID: 4e285f01-7d25-4c25-86bf-33f554e83b27, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Storage Domain iso (Data Center EECS-Primary) was deactivated by system because it's not visible by any of the hosts. 2017-10-01 15:28:06,927-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-2) [e5acb36] Removing vds '[]' from the domain in maintenance cache 2017-10-01 15:28:06,927-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-2) [e5acb36] Removing host(s) '[]' from hosts unseen domain report cache 2017-10-01 15:28:08,919-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [5532121c] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:28:08,919-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [5532121c] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 15:28:08,919-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [5532121c] HostName = virt2 2017-10-01 15:28:08,919-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [5532121c] Command 'ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:28:08,919-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [5532121c] FINISH, ConnectStoragePoolVDSCommand, log id: 18e0ca61 2017-10-01 15:28:08,919-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler5) [5532121c] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:28:09,013-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler5) [5532121c] IRS failover failed - can't allocate vds server 2017-10-01 15:28:10,995-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler10) [3f5c6de9] Command 'GetAllVmStatsVDSCommand(HostName = virt4, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846'})' execution failed: Recovering from crash or Initializing 2017-10-01 15:28:10,995-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.PollVmStatsRefresher] (DefaultQuartzScheduler10) [3f5c6de9] Failed to fetch vms info for host 'virt4' - skipping VMs monitoring. 2017-10-01 15:28:17,064-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [6f1b7bb8] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command GetStatsVDS failed: Recovering from crash or Initializing 2017-10-01 15:28:17,064-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStatsVDSCommand] (DefaultQuartzScheduler10) [6f1b7bb8] Command 'GetStatsVDSCommand(HostName = virt4, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vds='Host[virt4,5ae29ed2-9c0f-4754-8df3-b43d14726846]'})' execution failed: Recovering from crash or Initializing 2017-10-01 15:28:17,064-04 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [6f1b7bb8] Failed getting vds stats, host='virt4'(5ae29ed2-9c0f-4754-8df3-b43d14726846): org.ovirt.engine.core.vdsbroker.vdsbroker.VDSRecoveringException: Recovering from crash or Initializing 2017-10-01 15:28:19,110-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [6ca64da7] START, ConnectStoragePoolVDSCommand(HostName = virt1, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vdsId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 704b040f 2017-10-01 15:28:19,111-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [6ca64da7] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 15:28:26,022-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler4) [3486261] Command 'GetAllVmStatsVDSCommand(HostName = virt4, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846'})' execution failed: Recovering from crash or Initializing 2017-10-01 15:28:26,022-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.PollVmStatsRefresher] (DefaultQuartzScheduler4) [3486261] Failed to fetch vms info for host 'virt4' - skipping VMs monitoring. 2017-10-01 15:28:33,146-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [728f061a] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command GetStatsVDS failed: Recovering from crash or Initializing 2017-10-01 15:28:33,147-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStatsVDSCommand] (DefaultQuartzScheduler4) [728f061a] Command 'GetStatsVDSCommand(HostName = virt4, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vds='Host[virt4,5ae29ed2-9c0f-4754-8df3-b43d14726846]'})' execution failed: Recovering from crash or Initializing 2017-10-01 15:28:33,147-04 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [728f061a] Failed getting vds stats, host='virt4'(5ae29ed2-9c0f-4754-8df3-b43d14726846): org.ovirt.engine.core.vdsbroker.vdsbroker.VDSRecoveringException: Recovering from crash or Initializing 2017-10-01 15:28:41,050-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler7) [3fee3e5] Command 'GetAllVmStatsVDSCommand(HostName = virt4, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846'})' execution failed: Recovering from crash or Initializing 2017-10-01 15:28:41,050-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.PollVmStatsRefresher] (DefaultQuartzScheduler7) [3fee3e5] Failed to fetch vms info for host 'virt4' - skipping VMs monitoring. 2017-10-01 15:28:49,272-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [77d7ea3d] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command GetStatsVDS failed: Recovering from crash or Initializing 2017-10-01 15:28:49,272-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStatsVDSCommand] (DefaultQuartzScheduler4) [77d7ea3d] Command 'GetStatsVDSCommand(HostName = virt4, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vds='Host[virt4,5ae29ed2-9c0f-4754-8df3-b43d14726846]'})' execution failed: Recovering from crash or Initializing 2017-10-01 15:28:49,272-04 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [77d7ea3d] Failed getting vds stats, host='virt4'(5ae29ed2-9c0f-4754-8df3-b43d14726846): org.ovirt.engine.core.vdsbroker.vdsbroker.VDSRecoveringException: Recovering from crash or Initializing 2017-10-01 15:28:55,561-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [8d42125] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 15:28:56,075-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler9) [13e1194d] Command 'GetAllVmStatsVDSCommand(HostName = virt4, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846'})' execution failed: Recovering from crash or Initializing 2017-10-01 15:28:56,075-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.PollVmStatsRefresher] (DefaultQuartzScheduler9) [13e1194d] Failed to fetch vms info for host 'virt4' - skipping VMs monitoring. 2017-10-01 15:29:05,356-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [16f6ed97] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command GetStatsVDS failed: Recovering from crash or Initializing 2017-10-01 15:29:05,356-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStatsVDSCommand] (DefaultQuartzScheduler1) [16f6ed97] Command 'GetStatsVDSCommand(HostName = virt4, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vds='Host[virt4,5ae29ed2-9c0f-4754-8df3-b43d14726846]'})' execution failed: Recovering from crash or Initializing 2017-10-01 15:29:05,356-04 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [16f6ed97] Failed getting vds stats, host='virt4'(5ae29ed2-9c0f-4754-8df3-b43d14726846): org.ovirt.engine.core.vdsbroker.vdsbroker.VDSRecoveringException: Recovering from crash or Initializing 2017-10-01 15:29:11,100-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler4) [225e49b7] Command 'GetAllVmStatsVDSCommand(HostName = virt4, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846'})' execution failed: Recovering from crash or Initializing 2017-10-01 15:29:11,100-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.PollVmStatsRefresher] (DefaultQuartzScheduler4) [225e49b7] Failed to fetch vms info for host 'virt4' - skipping VMs monitoring. 2017-10-01 15:29:21,439-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [1dd51d5f] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command GetStatsVDS failed: Recovering from crash or Initializing 2017-10-01 15:29:21,439-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStatsVDSCommand] (DefaultQuartzScheduler3) [1dd51d5f] Command 'GetStatsVDSCommand(HostName = virt4, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vds='Host[virt4,5ae29ed2-9c0f-4754-8df3-b43d14726846]'})' execution failed: Recovering from crash or Initializing 2017-10-01 15:29:21,439-04 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [1dd51d5f] Failed getting vds stats, host='virt4'(5ae29ed2-9c0f-4754-8df3-b43d14726846): org.ovirt.engine.core.vdsbroker.vdsbroker.VDSRecoveringException: Recovering from crash or Initializing 2017-10-01 15:29:26,125-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler5) [86eab92] Command 'GetAllVmStatsVDSCommand(HostName = virt4, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846'})' execution failed: Recovering from crash or Initializing 2017-10-01 15:29:26,125-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.PollVmStatsRefresher] (DefaultQuartzScheduler5) [86eab92] Failed to fetch vms info for host 'virt4' - skipping VMs monitoring. 2017-10-01 15:29:28,464-04 ERROR [org.ovirt.engine.core.sso.servlets.InteractiveAuthServlet] (default task-237) [] Cannot authenticate user 'jas@eecs.yorku.ca': The username or password is incorrect. 2017-10-01 15:29:37,545-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [3d34e1f8] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command GetStatsVDS failed: Recovering from crash or Initializing 2017-10-01 15:29:37,545-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStatsVDSCommand] (DefaultQuartzScheduler8) [3d34e1f8] Command 'GetStatsVDSCommand(HostName = virt4, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vds='Host[virt4,5ae29ed2-9c0f-4754-8df3-b43d14726846]'})' execution failed: Recovering from crash or Initializing 2017-10-01 15:29:37,545-04 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [3d34e1f8] Failed getting vds stats, host='virt4'(5ae29ed2-9c0f-4754-8df3-b43d14726846): org.ovirt.engine.core.vdsbroker.vdsbroker.VDSRecoveringException: Recovering from crash or Initializing 2017-10-01 15:29:41,152-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler5) [62d9f87c] Command 'GetAllVmStatsVDSCommand(HostName = virt4, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846'})' execution failed: Recovering from crash or Initializing 2017-10-01 15:29:41,152-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.PollVmStatsRefresher] (DefaultQuartzScheduler5) [62d9f87c] Failed to fetch vms info for host 'virt4' - skipping VMs monitoring. 2017-10-01 15:29:53,629-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [36b03f63] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command GetStatsVDS failed: Recovering from crash or Initializing 2017-10-01 15:29:53,629-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStatsVDSCommand] (DefaultQuartzScheduler1) [36b03f63] Command 'GetStatsVDSCommand(HostName = virt4, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vds='Host[virt4,5ae29ed2-9c0f-4754-8df3-b43d14726846]'})' execution failed: Recovering from crash or Initializing 2017-10-01 15:29:53,629-04 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [36b03f63] Failed getting vds stats, host='virt4'(5ae29ed2-9c0f-4754-8df3-b43d14726846): org.ovirt.engine.core.vdsbroker.vdsbroker.VDSRecoveringException: Recovering from crash or Initializing 2017-10-01 15:29:55,582-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [705a8360] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 15:29:56,178-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler8) [2f4edc8a] Command 'GetAllVmStatsVDSCommand(HostName = virt4, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846'})' execution failed: Recovering from crash or Initializing 2017-10-01 15:29:56,178-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.PollVmStatsRefresher] (DefaultQuartzScheduler8) [2f4edc8a] Failed to fetch vms info for host 'virt4' - skipping VMs monitoring. 2017-10-01 15:29:56,630-04 INFO [org.ovirt.engine.extension.aaa.jdbc.core.Tasks] (default task-259) [] (house keeping) deleting failed logins prior to 2017-09-24 19:29:56Z. 2017-10-01 15:29:56,687-04 INFO [org.ovirt.engine.extension.aaa.jdbc.core.Tasks] (default task-259) [] (house keeping) deleting failed logins prior to 2017-09-24 19:29:56Z. 2017-10-01 15:29:56,722-04 INFO [org.ovirt.engine.core.sso.utils.AuthenticationUtils] (default task-259) [] User admin@internal successfully logged in with scopes: ovirt-app-admin ovirt-app-api ovirt-app-portal ovirt-ext=auth:sequence-priority=~ ovirt-ext=revoke:revoke-all ovirt-ext=token-info:authz-search ovirt-ext=token-info:public-authz-search ovirt-ext=token-info:validate ovirt-ext=token:password-access 2017-10-01 15:29:56,905-04 INFO [org.ovirt.engine.core.bll.aaa.CreateUserSessionCommand] (default task-258) [6dc086b6] Running command: CreateUserSessionCommand internal: false. 2017-10-01 15:29:56,915-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-258) [6dc086b6] EVENT_ID: USER_VDC_LOGIN(30), Correlation ID: 6dc086b6, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: User admin@internal logged in. 2017-10-01 15:30:00,025-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [64558b6a] Autorecovering 1 hosts 2017-10-01 15:30:00,025-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [64558b6a] Autorecovering hosts id: 5ae29ed2-9c0f-4754-8df3-b43d14726846 , name : virt4 2017-10-01 15:30:00,029-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [206fefea] Lock Acquired to object 'EngineLock:{exclusiveLocks='[5ae29ed2-9c0f-4754-8df3-b43d14726846=VDS]', sharedLocks=''}' 2017-10-01 15:30:00,047-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [206fefea] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 5ae29ed2-9c0f-4754-8df3-b43d14726846 Type: VDSAction group MANIPULATE_HOST with role type ADMIN 2017-10-01 15:30:00,047-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [206fefea] Before acquiring lock in order to prevent monitoring for host 'virt4' from data-center 'EECS-Primary' 2017-10-01 15:30:00,047-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [206fefea] Lock acquired, from now a monitoring of host will be skipped for host 'virt4' from data-center 'EECS-Primary' 2017-10-01 15:30:00,063-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [206fefea] START, SetVdsStatusVDSCommand(HostName = virt4, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 72647cb6 2017-10-01 15:30:00,065-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler8) [206fefea] FINISH, SetVdsStatusVDSCommand, log id: 72647cb6 2017-10-01 15:30:00,071-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [206fefea] Activate host finished. Lock released. Monitoring can run now for host 'virt4' from data-center 'EECS-Primary' 2017-10-01 15:30:00,090-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [206fefea] EVENT_ID: VDS_ACTIVATE_ASYNC(9,502), Correlation ID: 206fefea, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host virt4 was autorecovered. 2017-10-01 15:30:00,090-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler8) [206fefea] Lock freed to object 'EngineLock:{exclusiveLocks='[5ae29ed2-9c0f-4754-8df3-b43d14726846=VDS]', sharedLocks=''}' 2017-10-01 15:30:00,098-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [206fefea] Autorecovering 3 storage domains 2017-10-01 15:30:00,098-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [206fefea] Autorecovering storage domains id: c8653dba-63ec-4447-872c-0ffcc6af9c0f 2017-10-01 15:30:00,101-04 INFO [org.ovirt.engine.core.bll.storage.connection.ConnectDomainToStorageCommand] (DefaultQuartzScheduler8) [732a0e73] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: c8653dba-63ec-4447-872c-0ffcc6af9c0f Type: Storage 2017-10-01 15:30:00,101-04 INFO [org.ovirt.engine.core.bll.storage.connection.ConnectDomainToStorageCommand] (DefaultQuartzScheduler8) [732a0e73] ConnectDomainToStorage. Before Connect all hosts to pool. Time: Sun Oct 01 15:30:00 EDT 2017 2017-10-01 15:30:00,160-04 INFO [org.ovirt.engine.core.bll.storage.connection.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-6-thread-15) [736746f7] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN 2017-10-01 15:30:00,161-04 INFO [org.ovirt.engine.core.bll.storage.connection.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-6-thread-25) [8cfcf55] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN 2017-10-01 15:30:00,167-04 INFO [org.ovirt.engine.core.bll.storage.connection.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-6-thread-5) [4ea2d31d] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN 2017-10-01 15:30:00,182-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-25) [8cfcf55] START, ConnectStorageServerVDSCommand(HostName = virt3, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='00000000-0000-0000-0000-000000000000', storageType='NFS', connectionList='[StorageServerConnections:{id='5e557659-6817-40fa-9a55-7458125404a6', connection='virtfs-fs:/nfs/data', iqn='null', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 528ac806 2017-10-01 15:30:00,184-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-5) [4ea2d31d] START, ConnectStorageServerVDSCommand(HostName = virt2, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='00000000-0000-0000-0000-000000000000', storageType='NFS', connectionList='[StorageServerConnections:{id='5e557659-6817-40fa-9a55-7458125404a6', connection='virtfs-fs:/nfs/data', iqn='null', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 5258aa18 2017-10-01 15:30:00,197-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-15) [736746f7] START, ConnectStorageServerVDSCommand(HostName = virt1, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='00000000-0000-0000-0000-000000000000', storageType='NFS', connectionList='[StorageServerConnections:{id='5e557659-6817-40fa-9a55-7458125404a6', connection='virtfs-fs:/nfs/data', iqn='null', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 3f5d6c1 2017-10-01 15:30:03,682-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [41fcfa09] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command GetCapabilitiesVDS failed: Recovering from crash or Initializing 2017-10-01 15:30:03,682-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler1) [41fcfa09] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@25a48f46' 2017-10-01 15:30:03,682-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler1) [41fcfa09] HostName = virt4 2017-10-01 15:30:03,682-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler1) [41fcfa09] Command 'GetCapabilitiesVDSCommand(HostName = virt4, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vds='Host[virt4,5ae29ed2-9c0f-4754-8df3-b43d14726846]'})' execution failed: Recovering from crash or Initializing 2017-10-01 15:30:03,703-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [41fcfa09] EVENT_ID: VDS_INITIALIZING(514), Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSRecoveringException: Recovering from crash or Initializing 2017-10-01 15:30:03,703-04 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler1) [41fcfa09] Failed to refresh VDS, continuing, vds='virt4'(5ae29ed2-9c0f-4754-8df3-b43d14726846): Recovering from crash or Initializing 2017-10-01 15:30:07,727-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [58b4b63f] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command GetCapabilitiesVDS failed: Recovering from crash or Initializing 2017-10-01 15:30:07,727-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler3) [58b4b63f] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@4cb3313e' 2017-10-01 15:30:07,727-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler3) [58b4b63f] HostName = virt4 2017-10-01 15:30:07,727-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler3) [58b4b63f] Command 'GetCapabilitiesVDSCommand(HostName = virt4, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vds='Host[virt4,5ae29ed2-9c0f-4754-8df3-b43d14726846]'})' execution failed: Recovering from crash or Initializing 2017-10-01 15:30:09,857-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [6ca64da7] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt1 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:30:09,857-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [6ca64da7] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 15:30:09,857-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [6ca64da7] HostName = virt1 2017-10-01 15:30:09,857-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [6ca64da7] Command 'ConnectStoragePoolVDSCommand(HostName = virt1, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vdsId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:30:09,857-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [6ca64da7] FINISH, ConnectStoragePoolVDSCommand, log id: 704b040f 2017-10-01 15:30:09,857-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler10) [6ca64da7] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:30:11,183-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [676783bc] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command GetCapabilitiesVDS failed: Recovering from crash or Initializing 2017-10-01 15:30:11,183-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler7) [676783bc] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@613be238' 2017-10-01 15:30:11,183-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler7) [676783bc] HostName = virt4 2017-10-01 15:30:11,183-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler7) [676783bc] Command 'GetCapabilitiesVDSCommand(HostName = virt4, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vds='Host[virt4,5ae29ed2-9c0f-4754-8df3-b43d14726846]'})' execution failed: Recovering from crash or Initializing 2017-10-01 15:30:11,204-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler4) [77064ff5] Command 'GetAllVmStatsVDSCommand(HostName = virt4, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846'})' execution failed: Recovering from crash or Initializing 2017-10-01 15:30:11,204-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.PollVmStatsRefresher] (DefaultQuartzScheduler4) [77064ff5] Failed to fetch vms info for host 'virt4' - skipping VMs monitoring. 2017-10-01 15:30:15,210-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [6bdda234] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command GetCapabilitiesVDS failed: Recovering from crash or Initializing 2017-10-01 15:30:15,210-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler9) [6bdda234] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@240f5780' 2017-10-01 15:30:15,210-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler9) [6bdda234] HostName = virt4 2017-10-01 15:30:15,210-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler9) [6bdda234] Command 'GetCapabilitiesVDSCommand(HostName = virt4, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vds='Host[virt4,5ae29ed2-9c0f-4754-8df3-b43d14726846]'})' execution failed: Recovering from crash or Initializing 2017-10-01 15:30:19,233-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [6155c0e8] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command GetCapabilitiesVDS failed: Recovering from crash or Initializing 2017-10-01 15:30:19,233-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler5) [6155c0e8] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@4d8b508' 2017-10-01 15:30:19,233-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler5) [6155c0e8] HostName = virt4 2017-10-01 15:30:19,233-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler5) [6155c0e8] Command 'GetCapabilitiesVDSCommand(HostName = virt4, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vds='Host[virt4,5ae29ed2-9c0f-4754-8df3-b43d14726846]'})' execution failed: Recovering from crash or Initializing 2017-10-01 15:30:23,255-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6b9f4cef] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command GetCapabilitiesVDS failed: Recovering from crash or Initializing 2017-10-01 15:30:23,255-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler2) [6b9f4cef] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@2a68f043' 2017-10-01 15:30:23,255-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler2) [6b9f4cef] HostName = virt4 2017-10-01 15:30:23,255-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler2) [6b9f4cef] Command 'GetCapabilitiesVDSCommand(HostName = virt4, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vds='Host[virt4,5ae29ed2-9c0f-4754-8df3-b43d14726846]'})' execution failed: Recovering from crash or Initializing 2017-10-01 15:30:26,232-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler7) [1d8c0eee] Command 'GetAllVmStatsVDSCommand(HostName = virt4, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846'})' execution failed: Recovering from crash or Initializing 2017-10-01 15:30:26,232-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.PollVmStatsRefresher] (DefaultQuartzScheduler7) [1d8c0eee] Failed to fetch vms info for host 'virt4' - skipping VMs monitoring. 2017-10-01 15:30:26,288-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [623d23a] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command GetCapabilitiesVDS failed: Recovering from crash or Initializing 2017-10-01 15:30:26,288-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler1) [623d23a] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@3bc7341b' 2017-10-01 15:30:26,288-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler1) [623d23a] HostName = virt4 2017-10-01 15:30:26,288-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler1) [623d23a] Command 'GetCapabilitiesVDSCommand(HostName = virt4, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vds='Host[virt4,5ae29ed2-9c0f-4754-8df3-b43d14726846]'})' execution failed: Recovering from crash or Initializing 2017-10-01 15:30:30,309-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [2e882fd1] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command GetCapabilitiesVDS failed: Recovering from crash or Initializing 2017-10-01 15:30:30,310-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler7) [2e882fd1] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@69c4a561' 2017-10-01 15:30:30,310-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler7) [2e882fd1] HostName = virt4 2017-10-01 15:30:30,310-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler7) [2e882fd1] Command 'GetCapabilitiesVDSCommand(HostName = virt4, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vds='Host[virt4,5ae29ed2-9c0f-4754-8df3-b43d14726846]'})' execution failed: Recovering from crash or Initializing 2017-10-01 15:30:34,332-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [7451cd7e] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command GetCapabilitiesVDS failed: Recovering from crash or Initializing 2017-10-01 15:30:34,332-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler7) [7451cd7e] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@1e260509' 2017-10-01 15:30:34,332-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler7) [7451cd7e] HostName = virt4 2017-10-01 15:30:34,332-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler7) [7451cd7e] Command 'GetCapabilitiesVDSCommand(HostName = virt4, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vds='Host[virt4,5ae29ed2-9c0f-4754-8df3-b43d14726846]'})' execution failed: Recovering from crash or Initializing 2017-10-01 15:30:37,359-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [540e9884] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command GetCapabilitiesVDS failed: Recovering from crash or Initializing 2017-10-01 15:30:37,359-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler3) [540e9884] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@6b890f58' 2017-10-01 15:30:37,359-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler3) [540e9884] HostName = virt4 2017-10-01 15:30:37,359-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler3) [540e9884] Command 'GetCapabilitiesVDSCommand(HostName = virt4, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vds='Host[virt4,5ae29ed2-9c0f-4754-8df3-b43d14726846]'})' execution failed: Recovering from crash or Initializing 2017-10-01 15:30:41,161-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [1279eb4b] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command GetCapabilitiesVDS failed: Recovering from crash or Initializing 2017-10-01 15:30:41,161-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler9) [1279eb4b] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@4fdd34ad' 2017-10-01 15:30:41,161-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler9) [1279eb4b] HostName = virt4 2017-10-01 15:30:41,161-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler9) [1279eb4b] Command 'GetCapabilitiesVDSCommand(HostName = virt4, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vds='Host[virt4,5ae29ed2-9c0f-4754-8df3-b43d14726846]'})' execution failed: Recovering from crash or Initializing 2017-10-01 15:30:41,252-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler2) [6c7cc9e7] Command 'GetAllVmStatsVDSCommand(HostName = virt4, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846'})' execution failed: Recovering from crash or Initializing 2017-10-01 15:30:41,252-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.PollVmStatsRefresher] (DefaultQuartzScheduler2) [6c7cc9e7] Failed to fetch vms info for host 'virt4' - skipping VMs monitoring. 2017-10-01 15:30:45,184-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [759ee3c2] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command GetCapabilitiesVDS failed: Recovering from crash or Initializing 2017-10-01 15:30:45,184-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler2) [759ee3c2] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@22786fde' 2017-10-01 15:30:45,184-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler2) [759ee3c2] HostName = virt4 2017-10-01 15:30:45,184-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler2) [759ee3c2] Command 'GetCapabilitiesVDSCommand(HostName = virt4, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vds='Host[virt4,5ae29ed2-9c0f-4754-8df3-b43d14726846]'})' execution failed: Recovering from crash or Initializing 2017-10-01 15:30:49,207-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [647e3764] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command GetCapabilitiesVDS failed: Recovering from crash or Initializing 2017-10-01 15:30:49,207-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler9) [647e3764] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@661771cd' 2017-10-01 15:30:49,207-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler9) [647e3764] HostName = virt4 2017-10-01 15:30:49,207-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler9) [647e3764] Command 'GetCapabilitiesVDSCommand(HostName = virt4, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vds='Host[virt4,5ae29ed2-9c0f-4754-8df3-b43d14726846]'})' execution failed: Recovering from crash or Initializing 2017-10-01 15:30:53,229-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [406718fe] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command GetCapabilitiesVDS failed: Recovering from crash or Initializing 2017-10-01 15:30:53,229-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler1) [406718fe] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@55df475' 2017-10-01 15:30:53,229-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler1) [406718fe] HostName = virt4 2017-10-01 15:30:53,229-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler1) [406718fe] Command 'GetCapabilitiesVDSCommand(HostName = virt4, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vds='Host[virt4,5ae29ed2-9c0f-4754-8df3-b43d14726846]'})' execution failed: Recovering from crash or Initializing 2017-10-01 15:30:55,599-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [36fd9711] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 15:30:56,257-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [234ee06e] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command GetCapabilitiesVDS failed: Recovering from crash or Initializing 2017-10-01 15:30:56,257-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler4) [234ee06e] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@6943eb4d' 2017-10-01 15:30:56,257-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler4) [234ee06e] HostName = virt4 2017-10-01 15:30:56,257-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler4) [234ee06e] Command 'GetCapabilitiesVDSCommand(HostName = virt4, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vds='Host[virt4,5ae29ed2-9c0f-4754-8df3-b43d14726846]'})' execution failed: Recovering from crash or Initializing 2017-10-01 15:30:56,284-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler9) [7cf1dcba] Command 'GetAllVmStatsVDSCommand(HostName = virt4, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846'})' execution failed: Recovering from crash or Initializing 2017-10-01 15:30:56,284-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.PollVmStatsRefresher] (DefaultQuartzScheduler9) [7cf1dcba] Failed to fetch vms info for host 'virt4' - skipping VMs monitoring. 2017-10-01 15:31:00,201-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [7e709ecf] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command GetCapabilitiesVDS failed: Recovering from crash or Initializing 2017-10-01 15:31:00,201-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler1) [7e709ecf] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@45a03fb3' 2017-10-01 15:31:00,201-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler1) [7e709ecf] HostName = virt4 2017-10-01 15:31:00,201-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler1) [7e709ecf] Command 'GetCapabilitiesVDSCommand(HostName = virt4, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vds='Host[virt4,5ae29ed2-9c0f-4754-8df3-b43d14726846]'})' execution failed: Recovering from crash or Initializing 2017-10-01 15:31:00,207-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-25) [8cfcf55] FINISH, ConnectStorageServerVDSCommand, return: {5e557659-6817-40fa-9a55-7458125404a6=0}, log id: 528ac806 2017-10-01 15:31:00,212-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-5) [4ea2d31d] FINISH, ConnectStorageServerVDSCommand, return: {5e557659-6817-40fa-9a55-7458125404a6=0}, log id: 5258aa18 2017-10-01 15:31:01,214-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-15) [736746f7] FINISH, ConnectStorageServerVDSCommand, return: {5e557659-6817-40fa-9a55-7458125404a6=0}, log id: 3f5d6c1 2017-10-01 15:31:01,218-04 INFO [org.ovirt.engine.core.bll.storage.connection.ConnectDomainToStorageCommand] (DefaultQuartzScheduler8) [732a0e73] ConnectDomainToStorage. After Connect all hosts to pool. Time: Sun Oct 01 15:31:01 EDT 2017 2017-10-01 15:31:01,219-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [732a0e73] Autorecovering storage domains id: f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2 2017-10-01 15:31:01,223-04 INFO [org.ovirt.engine.core.bll.storage.connection.ConnectDomainToStorageCommand] (DefaultQuartzScheduler8) [2f69566e] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2 Type: Storage 2017-10-01 15:31:01,223-04 INFO [org.ovirt.engine.core.bll.storage.connection.ConnectDomainToStorageCommand] (DefaultQuartzScheduler8) [2f69566e] ConnectDomainToStorage. Before Connect all hosts to pool. Time: Sun Oct 01 15:31:01 EDT 2017 2017-10-01 15:31:01,283-04 INFO [org.ovirt.engine.core.bll.storage.connection.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-6-thread-48) [1cf6d715] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN 2017-10-01 15:31:01,283-04 INFO [org.ovirt.engine.core.bll.storage.connection.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-6-thread-35) [7e83fd9d] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN 2017-10-01 15:31:01,283-04 INFO [org.ovirt.engine.core.bll.storage.connection.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-6-thread-50) [3ae1769c] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN 2017-10-01 15:31:01,298-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-48) [1cf6d715] START, ConnectStorageServerVDSCommand(HostName = virt1, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='00000000-0000-0000-0000-000000000000', storageType='NFS', connectionList='[StorageServerConnections:{id='417fdd51-dbcd-41d9-9a05-ef4d9f485d13', connection='virtfs-fs:/nfs/export', iqn='null', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 59bce067 2017-10-01 15:31:01,298-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-50) [3ae1769c] START, ConnectStorageServerVDSCommand(HostName = virt2, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='00000000-0000-0000-0000-000000000000', storageType='NFS', connectionList='[StorageServerConnections:{id='417fdd51-dbcd-41d9-9a05-ef4d9f485d13', connection='virtfs-fs:/nfs/export', iqn='null', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 74f856d5 2017-10-01 15:31:01,298-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-35) [7e83fd9d] START, ConnectStorageServerVDSCommand(HostName = virt3, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='00000000-0000-0000-0000-000000000000', storageType='NFS', connectionList='[StorageServerConnections:{id='417fdd51-dbcd-41d9-9a05-ef4d9f485d13', connection='virtfs-fs:/nfs/export', iqn='null', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 6ebbd566 2017-10-01 15:31:04,224-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [189a1c10] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command GetCapabilitiesVDS failed: Recovering from crash or Initializing 2017-10-01 15:31:04,225-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler9) [189a1c10] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@396e5a06' 2017-10-01 15:31:04,225-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler9) [189a1c10] HostName = virt4 2017-10-01 15:31:04,225-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler9) [189a1c10] Command 'GetCapabilitiesVDSCommand(HostName = virt4, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vds='Host[virt4,5ae29ed2-9c0f-4754-8df3-b43d14726846]'})' execution failed: Recovering from crash or Initializing 2017-10-01 15:31:08,247-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6aea1c67] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command GetCapabilitiesVDS failed: Recovering from crash or Initializing 2017-10-01 15:31:08,247-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler2) [6aea1c67] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@4ab28be3' 2017-10-01 15:31:08,247-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler2) [6aea1c67] HostName = virt4 2017-10-01 15:31:08,247-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler2) [6aea1c67] Command 'GetCapabilitiesVDSCommand(HostName = virt4, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vds='Host[virt4,5ae29ed2-9c0f-4754-8df3-b43d14726846]'})' execution failed: Recovering from crash or Initializing 2017-10-01 15:31:10,698-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [6ca64da7] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt1 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 15:31:10,698-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler10) [6ca64da7] Command 'SpmStatusVDSCommand(HostName = virt1, SpmStatusVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 15:31:10,698-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler10) [6ca64da7] Irs placed on server '45f8b331-842e-48e7-9df8-56adddb93836' failed. Proceed Failover 2017-10-01 15:31:10,756-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [6ca64da7] START, ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 5784f5c9 2017-10-01 15:31:10,756-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [6ca64da7] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 15:31:11,289-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [4b7a3f08] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command GetCapabilitiesVDS failed: Recovering from crash or Initializing 2017-10-01 15:31:11,289-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler3) [4b7a3f08] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@7c0ddbeb' 2017-10-01 15:31:11,289-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler3) [4b7a3f08] HostName = virt4 2017-10-01 15:31:11,289-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler3) [4b7a3f08] Command 'GetCapabilitiesVDSCommand(HostName = virt4, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vds='Host[virt4,5ae29ed2-9c0f-4754-8df3-b43d14726846]'})' execution failed: Recovering from crash or Initializing 2017-10-01 15:31:11,310-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler4) [5e5e9aca] Command 'GetAllVmStatsVDSCommand(HostName = virt4, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846'})' execution failed: Recovering from crash or Initializing 2017-10-01 15:31:11,310-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.PollVmStatsRefresher] (DefaultQuartzScheduler4) [5e5e9aca] Failed to fetch vms info for host 'virt4' - skipping VMs monitoring. 2017-10-01 15:31:15,312-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [24ac38e0] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command GetCapabilitiesVDS failed: Recovering from crash or Initializing 2017-10-01 15:31:15,312-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler7) [24ac38e0] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@472e03b0' 2017-10-01 15:31:15,312-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler7) [24ac38e0] HostName = virt4 2017-10-01 15:31:15,312-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler7) [24ac38e0] Command 'GetCapabilitiesVDSCommand(HostName = virt4, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vds='Host[virt4,5ae29ed2-9c0f-4754-8df3-b43d14726846]'})' execution failed: Recovering from crash or Initializing 2017-10-01 15:31:19,361-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [46c4259b] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command GetCapabilitiesVDS failed: Recovering from crash or Initializing 2017-10-01 15:31:19,361-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler3) [46c4259b] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@2335cdd7' 2017-10-01 15:31:19,361-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler3) [46c4259b] HostName = virt4 2017-10-01 15:31:19,361-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler3) [46c4259b] Command 'GetCapabilitiesVDSCommand(HostName = virt4, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vds='Host[virt4,5ae29ed2-9c0f-4754-8df3-b43d14726846]'})' execution failed: Recovering from crash or Initializing 2017-10-01 15:31:23,383-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [64457bf9] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command GetCapabilitiesVDS failed: Recovering from crash or Initializing 2017-10-01 15:31:23,383-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler1) [64457bf9] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@2aba2c8c' 2017-10-01 15:31:23,383-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler1) [64457bf9] HostName = virt4 2017-10-01 15:31:23,383-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler1) [64457bf9] Command 'GetCapabilitiesVDSCommand(HostName = virt4, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vds='Host[virt4,5ae29ed2-9c0f-4754-8df3-b43d14726846]'})' execution failed: Recovering from crash or Initializing 2017-10-01 15:31:26,337-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler9) [3f8bd1ee] Command 'GetAllVmStatsVDSCommand(HostName = virt4, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846'})' execution failed: Recovering from crash or Initializing 2017-10-01 15:31:26,337-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.PollVmStatsRefresher] (DefaultQuartzScheduler9) [3f8bd1ee] Failed to fetch vms info for host 'virt4' - skipping VMs monitoring. 2017-10-01 15:31:26,806-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [3defe2fd] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command GetCapabilitiesVDS failed: Recovering from crash or Initializing 2017-10-01 15:31:26,806-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler4) [3defe2fd] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@36fb2361' 2017-10-01 15:31:26,806-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler4) [3defe2fd] HostName = virt4 2017-10-01 15:31:26,806-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler4) [3defe2fd] Command 'GetCapabilitiesVDSCommand(HostName = virt4, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vds='Host[virt4,5ae29ed2-9c0f-4754-8df3-b43d14726846]'})' execution failed: Recovering from crash or Initializing 2017-10-01 15:31:30,828-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7ed3e666] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command GetCapabilitiesVDS failed: Recovering from crash or Initializing 2017-10-01 15:31:30,828-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler2) [7ed3e666] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@67b586ab' 2017-10-01 15:31:30,828-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler2) [7ed3e666] HostName = virt4 2017-10-01 15:31:30,828-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler2) [7ed3e666] Command 'GetCapabilitiesVDSCommand(HostName = virt4, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vds='Host[virt4,5ae29ed2-9c0f-4754-8df3-b43d14726846]'})' execution failed: Recovering from crash or Initializing 2017-10-01 15:31:34,851-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [2be73473] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command GetCapabilitiesVDS failed: Recovering from crash or Initializing 2017-10-01 15:31:34,851-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler3) [2be73473] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@2c61eae3' 2017-10-01 15:31:34,851-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler3) [2be73473] HostName = virt4 2017-10-01 15:31:34,851-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler3) [2be73473] Command 'GetCapabilitiesVDSCommand(HostName = virt4, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vds='Host[virt4,5ae29ed2-9c0f-4754-8df3-b43d14726846]'})' execution failed: Recovering from crash or Initializing 2017-10-01 15:31:38,873-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [9187556] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command GetCapabilitiesVDS failed: Recovering from crash or Initializing 2017-10-01 15:31:38,873-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler6) [9187556] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@3c8eab51' 2017-10-01 15:31:38,873-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler6) [9187556] HostName = virt4 2017-10-01 15:31:38,873-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler6) [9187556] Command 'GetCapabilitiesVDSCommand(HostName = virt4, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vds='Host[virt4,5ae29ed2-9c0f-4754-8df3-b43d14726846]'})' execution failed: Recovering from crash or Initializing 2017-10-01 15:31:41,366-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler7) [2db56e6e] Command 'GetAllVmStatsVDSCommand(HostName = virt4, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846'})' execution failed: Recovering from crash or Initializing 2017-10-01 15:31:41,366-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.PollVmStatsRefresher] (DefaultQuartzScheduler7) [2db56e6e] Failed to fetch vms info for host 'virt4' - skipping VMs monitoring. 2017-10-01 15:31:42,896-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [1d22529f] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command GetCapabilitiesVDS failed: Recovering from crash or Initializing 2017-10-01 15:31:42,896-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler2) [1d22529f] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@79044139' 2017-10-01 15:31:42,896-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler2) [1d22529f] HostName = virt4 2017-10-01 15:31:42,896-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler2) [1d22529f] Command 'GetCapabilitiesVDSCommand(HostName = virt4, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vds='Host[virt4,5ae29ed2-9c0f-4754-8df3-b43d14726846]'})' execution failed: Recovering from crash or Initializing 2017-10-01 15:31:46,919-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [7740886a] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command GetCapabilitiesVDS failed: Recovering from crash or Initializing 2017-10-01 15:31:46,919-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler2) [7740886a] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@22ca14ad' 2017-10-01 15:31:46,919-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler2) [7740886a] HostName = virt4 2017-10-01 15:31:46,919-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler2) [7740886a] Command 'GetCapabilitiesVDSCommand(HostName = virt4, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vds='Host[virt4,5ae29ed2-9c0f-4754-8df3-b43d14726846]'})' execution failed: Recovering from crash or Initializing 2017-10-01 15:31:50,942-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [6d360d2] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command GetCapabilitiesVDS failed: Recovering from crash or Initializing 2017-10-01 15:31:50,942-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler2) [6d360d2] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@2dd75108' 2017-10-01 15:31:50,942-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler2) [6d360d2] HostName = virt4 2017-10-01 15:31:50,942-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler2) [6d360d2] Command 'GetCapabilitiesVDSCommand(HostName = virt4, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vds='Host[virt4,5ae29ed2-9c0f-4754-8df3-b43d14726846]'})' execution failed: Recovering from crash or Initializing 2017-10-01 15:31:54,780-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [76439e7e] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command GetCapabilitiesVDS failed: Recovering from crash or Initializing 2017-10-01 15:31:54,780-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler9) [76439e7e] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@20eb3632' 2017-10-01 15:31:54,780-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler9) [76439e7e] HostName = virt4 2017-10-01 15:31:54,780-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler9) [76439e7e] Command 'GetCapabilitiesVDSCommand(HostName = virt4, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vds='Host[virt4,5ae29ed2-9c0f-4754-8df3-b43d14726846]'})' execution failed: Recovering from crash or Initializing 2017-10-01 15:31:54,811-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-50) [3ae1769c] FINISH, ConnectStorageServerVDSCommand, return: {417fdd51-dbcd-41d9-9a05-ef4d9f485d13=0}, log id: 74f856d5 2017-10-01 15:31:54,815-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-48) [1cf6d715] FINISH, ConnectStorageServerVDSCommand, return: {417fdd51-dbcd-41d9-9a05-ef4d9f485d13=0}, log id: 59bce067 2017-10-01 15:31:55,619-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [69fd4cc1] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 15:31:55,874-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [6ca64da7] FINISH, ConnectStoragePoolVDSCommand, log id: 5784f5c9 2017-10-01 15:31:56,367-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler10) [6ca64da7] hostFromVds::selectedVds - 'virt2', spmStatus 'Free', storage pool 'EECS-Primary', storage pool version '4.1' 2017-10-01 15:31:56,369-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler10) [6ca64da7] SPM Init: could not find reported vds or not up - pool: 'EECS-Primary' vds_spm_id: '4' 2017-10-01 15:31:56,384-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler10) [6ca64da7] SPM selection - vds seems as spm 'virt4' 2017-10-01 15:31:56,391-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler6) [66183434] Command 'GetAllVmStatsVDSCommand(HostName = virt4, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846'})' execution failed: Recovering from crash or Initializing 2017-10-01 15:31:56,391-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.PollVmStatsRefresher] (DefaultQuartzScheduler6) [66183434] Failed to fetch vms info for host 'virt4' - skipping VMs monitoring. 2017-10-01 15:31:56,398-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (DefaultQuartzScheduler10) [6ca64da7] START, SpmStopVDSCommand(HostName = virt4, SpmStopVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'}), log id: 2f227d4a 2017-10-01 15:31:56,410-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [6ca64da7] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command HSMGetAllTasksStatusesVDS failed: Recovering from crash or Initializing 2017-10-01 15:31:56,410-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler10) [6ca64da7] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand' return value ' 2017-10-01 15:31:56,410-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler10) [6ca64da7] HostName = virt4 2017-10-01 15:31:56,410-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler10) [6ca64da7] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = virt4, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846'})' execution failed: Recovering from crash or Initializing 2017-10-01 15:31:56,410-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (DefaultQuartzScheduler10) [6ca64da7] SpmStopVDSCommand::Stopping SPM on vds 'virt4', pool id 'aca61eac-ed8a-484b-bbee-3645148f6405' 2017-10-01 15:31:56,434-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [6ca64da7] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command SpmStopVDS failed: Recovering from crash or Initializing 2017-10-01 15:31:56,434-04 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (DefaultQuartzScheduler10) [6ca64da7] Could not stop spm of pool 'aca61eac-ed8a-484b-bbee-3645148f6405' on vds '5ae29ed2-9c0f-4754-8df3-b43d14726846': Recovering from crash or Initializing 2017-10-01 15:31:56,434-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (DefaultQuartzScheduler10) [6ca64da7] FINISH, SpmStopVDSCommand, log id: 2f227d4a 2017-10-01 15:31:56,434-04 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler10) [6ca64da7] spm stop on spm failed, stopping spm selection! 2017-10-01 15:31:56,442-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler5) [655048a2] VM 'db3e08fb-c111-442d-aeec-1507da30f486'(print) moved from 'NotResponding' --> 'Up' 2017-10-01 15:31:56,443-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler5) [655048a2] VM 'ae56c31c-71f7-433a-b198-a1c2c846325e'(forum) moved from 'NotResponding' --> 'Up' 2017-10-01 15:31:56,443-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler5) [655048a2] VM 'ddf37076-e795-4c87-a4fb-9efcaae76d72'(dc1) moved from 'NotResponding' --> 'Up' 2017-10-01 15:31:56,443-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler5) [655048a2] VM 'bfe9a524-2238-4bab-a91f-5dc6ed270896'(pcmode) moved from 'NotResponding' --> 'Up' 2017-10-01 15:31:56,444-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler5) [655048a2] VM '1325cebf-834e-4571-a3e1-82beb0021ffe'(dns1) moved from 'NotResponding' --> 'Up' 2017-10-01 15:31:56,444-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler5) [655048a2] VM 'a82b4cc3-6ef6-4a8e-97ef-1d4ea1b7e07c'(indigo) moved from 'NotResponding' --> 'Up' 2017-10-01 15:31:56,444-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler5) [655048a2] VM 'c8682859-9e55-460a-9ef0-d03e4a85f50e'(dhcp1) moved from 'NotResponding' --> 'Up' 2017-10-01 15:31:56,445-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler5) [655048a2] VM '3bb83ad5-dc00-445d-8708-2db18c53e0e6'(www-vhost) moved from 'NotResponding' --> 'Up' 2017-10-01 15:31:56,445-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler5) [655048a2] VM 'a8d4e149-55ec-4961-a414-d7d1e1b05a59'(fix) moved from 'NotResponding' --> 'Up' 2017-10-01 15:31:56,446-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler5) [655048a2] VM 'aa487207-7ff4-465a-9d9b-2a103d50dc77'(ftp1) moved from 'NotResponding' --> 'Up' 2017-10-01 15:31:56,446-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler5) [655048a2] VM '4ba80442-b361-4026-ae8c-895fd28042b0'(dist) moved from 'NotResponding' --> 'Up' 2017-10-01 15:31:56,456-04 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler3) [525e683d] Storage Domain 'f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2:export' was reported by Host 'virt1' as Active in Pool 'aca61eac-ed8a-484b-bbee-3645148f6405', moving to active status 2017-10-01 15:31:59,016-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [648b475b] START, GetHardwareInfoVDSCommand(HostName = virt4, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vds='Host[virt4,5ae29ed2-9c0f-4754-8df3-b43d14726846]'}), log id: 65b94bef 2017-10-01 15:31:59,020-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [648b475b] FINISH, GetHardwareInfoVDSCommand, log id: 65b94bef 2017-10-01 15:31:59,114-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [383b1747] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5ae29ed2-9c0f-4754-8df3-b43d14726846 Type: VDS 2017-10-01 15:31:59,136-04 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler10) [383b1747] Cancelling the recovery from crash timer for VDS '5ae29ed2-9c0f-4754-8df3-b43d14726846' because vds started initializing 2017-10-01 15:31:59,140-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [52bfeaf5] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: aca61eac-ed8a-484b-bbee-3645148f6405 Type: StoragePool 2017-10-01 15:31:59,146-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [27e03dc1] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: aca61eac-ed8a-484b-bbee-3645148f6405 Type: StoragePool 2017-10-01 15:31:59,216-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [27e03dc1] START, ConnectStorageServerVDSCommand(HostName = virt4, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', storageType='NFS', connectionList='[StorageServerConnections:{id='5e557659-6817-40fa-9a55-7458125404a6', connection='virtfs-fs:/nfs/data', iqn='null', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='13cf1e67-9961-4a25-a41f-cc76b4ab793d', connection='virtfs-fs:/nfs/iso', iqn='null', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='417fdd51-dbcd-41d9-9a05-ef4d9f485d13', connection='virtfs-fs:/nfs/export', iqn='null', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 7f72de25 2017-10-01 15:32:00,043-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [27e03dc1] FINISH, ConnectStorageServerVDSCommand, return: {5e557659-6817-40fa-9a55-7458125404a6=0, 417fdd51-dbcd-41d9-9a05-ef4d9f485d13=0, 13cf1e67-9961-4a25-a41f-cc76b4ab793d=0}, log id: 7f72de25 2017-10-01 15:32:00,047-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [27e03dc1] Host 'virt4' storage connection was succeeded 2017-10-01 15:32:00,057-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] START, ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 5797b467 2017-10-01 15:32:00,057-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 15:32:00,250-04 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler5) [1961a89f] Storage Domain 'c8653dba-63ec-4447-872c-0ffcc6af9c0f:data' was reported by Host 'virt2' as Active in Pool 'aca61eac-ed8a-484b-bbee-3645148f6405', moving to active status 2017-10-01 15:32:00,262-04 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler5) [1961a89f] Storage Domain '63d680c7-4c94-4b29-a6cb-f288a128b9bc:iso' was reported by Host 'virt2' as Active in Pool 'aca61eac-ed8a-484b-bbee-3645148f6405', moving to active status 2017-10-01 15:32:01,243-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-32) [] FINISH, ConnectStoragePoolVDSCommand, log id: 5797b467 2017-10-01 15:32:01,262-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-35) [7e83fd9d] FINISH, ConnectStorageServerVDSCommand, return: {417fdd51-dbcd-41d9-9a05-ef4d9f485d13=0}, log id: 6ebbd566 2017-10-01 15:32:01,265-04 INFO [org.ovirt.engine.core.bll.storage.connection.ConnectDomainToStorageCommand] (DefaultQuartzScheduler8) [2f69566e] ConnectDomainToStorage. After Connect all hosts to pool. Time: Sun Oct 01 15:32:01 EDT 2017 2017-10-01 15:32:01,266-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler8) [2f69566e] Autorecovering storage domains id: 63d680c7-4c94-4b29-a6cb-f288a128b9bc 2017-10-01 15:32:01,270-04 INFO [org.ovirt.engine.core.bll.storage.connection.ConnectDomainToStorageCommand] (DefaultQuartzScheduler8) [3a4af42] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 63d680c7-4c94-4b29-a6cb-f288a128b9bc Type: Storage 2017-10-01 15:32:01,270-04 INFO [org.ovirt.engine.core.bll.storage.connection.ConnectDomainToStorageCommand] (DefaultQuartzScheduler8) [3a4af42] ConnectDomainToStorage. Before Connect all hosts to pool. Time: Sun Oct 01 15:32:01 EDT 2017 2017-10-01 15:32:01,322-04 INFO [org.ovirt.engine.core.bll.storage.connection.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-6-thread-24) [437f1bf] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN 2017-10-01 15:32:01,322-04 INFO [org.ovirt.engine.core.bll.storage.connection.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-6-thread-44) [a62a3c9] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN 2017-10-01 15:32:01,322-04 INFO [org.ovirt.engine.core.bll.storage.connection.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-6-thread-40) [2e378fe3] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN 2017-10-01 15:32:01,338-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-24) [437f1bf] START, ConnectStorageServerVDSCommand(HostName = virt2, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='00000000-0000-0000-0000-000000000000', storageType='NFS', connectionList='[StorageServerConnections:{id='13cf1e67-9961-4a25-a41f-cc76b4ab793d', connection='virtfs-fs:/nfs/iso', iqn='null', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 6756bb1b 2017-10-01 15:32:01,338-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-40) [2e378fe3] START, ConnectStorageServerVDSCommand(HostName = virt3, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='00000000-0000-0000-0000-000000000000', storageType='NFS', connectionList='[StorageServerConnections:{id='13cf1e67-9961-4a25-a41f-cc76b4ab793d', connection='virtfs-fs:/nfs/iso', iqn='null', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 15f63a9d 2017-10-01 15:32:01,345-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-24) [437f1bf] FINISH, ConnectStorageServerVDSCommand, return: {13cf1e67-9961-4a25-a41f-cc76b4ab793d=0}, log id: 6756bb1b 2017-10-01 15:32:01,359-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-44) [a62a3c9] START, ConnectStorageServerVDSCommand(HostName = virt1, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='00000000-0000-0000-0000-000000000000', storageType='NFS', connectionList='[StorageServerConnections:{id='13cf1e67-9961-4a25-a41f-cc76b4ab793d', connection='virtfs-fs:/nfs/iso', iqn='null', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 699c33e9 2017-10-01 15:32:01,378-04 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-9) [] domain '63d680c7-4c94-4b29-a6cb-f288a128b9bc:iso' in problem 'STORAGE_ACCCESS_ERROR'. vds: 'virt3' 2017-10-01 15:32:01,378-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-9) [] Host 'e1e71e9d-a0cd-47db-96c6-f724f6cea736' has reported new storage access problem to the following domains '63d680c7-4c94-4b29-a6cb-f288a128b9bc' marking it for storage connections and pool metadata refresh (report id: 'ecf7ba6a-babd-4772-9dee-e6eba39645ec') 2017-10-01 15:32:01,510-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [27e03dc1] EVENT_ID: FENCE_OPERATION_USING_AGENT_AND_PROXY_STARTED(9,020), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Executing power management status on Host virt4 using Proxy Host virt1 and Fence Agent ipmilan:virt4-idrac. 2017-10-01 15:32:01,510-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler10) [27e03dc1] START, FenceVdsVDSCommand(HostName = virt1, FenceVdsVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', targetVdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', action='STATUS', agent='FenceAgent:{id='6a1fc1bb-0358-475a-aed2-e9c1868a2fc8', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', order='1', type='ipmilan', ip='virt4-idrac', port='null', user='root', password='***', encryptOptions='false', options='privlvl=OPERATOR, delay=10, lanplus=1'}', policy='null'}), log id: 419e2b78 2017-10-01 15:32:01,512-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-44) [a62a3c9] FINISH, ConnectStorageServerVDSCommand, return: {13cf1e67-9961-4a25-a41f-cc76b4ab793d=0}, log id: 699c33e9 2017-10-01 15:32:01,638-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-40) [2e378fe3] FINISH, ConnectStorageServerVDSCommand, return: {13cf1e67-9961-4a25-a41f-cc76b4ab793d=0}, log id: 15f63a9d 2017-10-01 15:32:01,641-04 INFO [org.ovirt.engine.core.bll.storage.connection.ConnectDomainToStorageCommand] (DefaultQuartzScheduler8) [3a4af42] ConnectDomainToStorage. After Connect all hosts to pool. Time: Sun Oct 01 15:32:01 EDT 2017 2017-10-01 15:32:02,635-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler10) [27e03dc1] FINISH, FenceVdsVDSCommand, return: FenceOperationResult:{status='SUCCESS', powerStatus='ON', message=''}, log id: 419e2b78 2017-10-01 15:32:02,650-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler10) [27e03dc1] START, SetMOMPolicyParametersVDSCommand(HostName = virt4, MomPolicyVDSParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846'}), log id: 6d38e426 2017-10-01 15:32:03,688-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler10) [27e03dc1] FINISH, SetMOMPolicyParametersVDSCommand, log id: 6d38e426 2017-10-01 15:32:03,707-04 INFO [org.ovirt.engine.core.bll.hostdev.RefreshHostDevicesCommand] (DefaultQuartzScheduler10) [52fe95bf] Running command: RefreshHostDevicesCommand internal: true. Entities affected : ID: 5ae29ed2-9c0f-4754-8df3-b43d14726846 Type: VDSAction group MANIPULATE_HOST with role type ADMIN 2017-10-01 15:32:03,976-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [52fe95bf] EVENT_ID: VDS_DETECTED(13), Correlation ID: 52bfeaf5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Status of host virt4 was set to Up. 2017-10-01 15:32:03,979-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [52fe95bf] EVENT_ID: VDS_FENCE_STATUS(496), Correlation ID: 52bfeaf5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host virt4 power management was verified successfully. 2017-10-01 15:32:04,013-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [215e3a15] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5ae29ed2-9c0f-4754-8df3-b43d14726846 Type: VDS 2017-10-01 15:32:06,626-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [39bd9aba] START, ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 5c3dd0b1 2017-10-01 15:32:06,626-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [39bd9aba] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 15:32:06,674-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler6) [4041908d] Running storage connections refresh for hosts '[e1e71e9d-a0cd-47db-96c6-f724f6cea736]' 2017-10-01 15:32:06,678-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (org.ovirt.thread.pool-6-thread-26) [228efa13] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: aca61eac-ed8a-484b-bbee-3645148f6405 Type: StoragePool 2017-10-01 15:32:06,693-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-26) [228efa13] START, ConnectStorageServerVDSCommand(HostName = virt3, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', storageType='NFS', connectionList='[StorageServerConnections:{id='5e557659-6817-40fa-9a55-7458125404a6', connection='virtfs-fs:/nfs/data', iqn='null', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='13cf1e67-9961-4a25-a41f-cc76b4ab793d', connection='virtfs-fs:/nfs/iso', iqn='null', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='417fdd51-dbcd-41d9-9a05-ef4d9f485d13', connection='virtfs-fs:/nfs/export', iqn='null', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 214e0fe6 2017-10-01 15:32:06,770-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-26) [228efa13] FINISH, ConnectStorageServerVDSCommand, return: {5e557659-6817-40fa-9a55-7458125404a6=0, 417fdd51-dbcd-41d9-9a05-ef4d9f485d13=0, 13cf1e67-9961-4a25-a41f-cc76b4ab793d=0}, log id: 214e0fe6 2017-10-01 15:32:06,774-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (org.ovirt.thread.pool-6-thread-26) [228efa13] Host 'virt3' storage connection was succeeded 2017-10-01 15:32:06,775-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler6) [4041908d] Submitting to the event queue pool refresh for hosts '[e1e71e9d-a0cd-47db-96c6-f724f6cea736]' 2017-10-01 15:32:06,776-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-13) [] Running storage pool metadata refresh for hosts '[e1e71e9d-a0cd-47db-96c6-f724f6cea736]' 2017-10-01 15:32:06,776-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-33) [] START, ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 2a520c93 2017-10-01 15:32:06,776-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-33) [] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 15:32:06,913-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [39bd9aba] FINISH, ConnectStoragePoolVDSCommand, log id: 5c3dd0b1 2017-10-01 15:32:06,925-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-33) [] FINISH, ConnectStoragePoolVDSCommand, log id: 2a520c93 2017-10-01 15:32:08,927-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler7) [39bd9aba] hostFromVds::selectedVds - 'virt3', spmStatus 'Free', storage pool 'EECS-Primary', storage pool version '4.1' 2017-10-01 15:32:08,928-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler7) [39bd9aba] Found spm host '5ae29ed2-9c0f-4754-8df3-b43d14726846', host name: 'virt4', according to spmId: '4'. 2017-10-01 15:32:09,936-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler7) [39bd9aba] SpmStatus on vds '5ae29ed2-9c0f-4754-8df3-b43d14726846': 'Free' 2017-10-01 15:32:09,938-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler7) [39bd9aba] starting spm on vds 'virt3', storage pool 'EECS-Primary', prevId '4', LVER '59' 2017-10-01 15:32:09,938-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler7) [39bd9aba] START, SpmStartVDSCommand(HostName = virt3, SpmStartVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', prevId='4', prevLVER='59', storagePoolFormatType='V4', recoveryMode='Manual', SCSIFencing='false'}), log id: 5d7f43da 2017-10-01 15:32:10,065-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler7) [39bd9aba] spmStart polling started: taskId 'cc096f6f-7c74-4c60-97ba-62914e10f68d' 2017-10-01 15:32:11,416-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler2) [294ed6ab] VM 'bcd5776e-978b-4723-933d-70a80322a796'(cape) moved from 'NotResponding' --> 'Up' 2017-10-01 15:32:11,416-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler2) [294ed6ab] VM '3e5a5728-8940-4fe7-b4af-11fb8e12f116'(www) moved from 'NotResponding' --> 'Up' 2017-10-01 15:32:11,416-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler2) [294ed6ab] VM 'f87ec08d-0fdf-42ba-8aac-c66c404d7aae'(webapp) moved from 'NotResponding' --> 'Up' 2017-10-01 15:32:11,416-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler2) [294ed6ab] VM '911d6883-607c-464e-b46b-8e6141bdd2de'(hdb) moved from 'NotResponding' --> 'Up' 2017-10-01 15:32:11,416-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler2) [294ed6ab] VM '8391f9a7-7fe7-4025-8fba-6273ae587a52'(archive) moved from 'NotResponding' --> 'Up' 2017-10-01 15:32:11,417-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler2) [294ed6ab] VM '1c7e966f-29e8-4788-b64c-86e52f415061'(udb) moved from 'NotResponding' --> 'Up' 2017-10-01 15:32:11,417-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler2) [294ed6ab] VM 'd66235b4-7c8d-4da2-a572-82ecc9c53a59'(www-las) moved from 'NotResponding' --> 'Up' 2017-10-01 15:32:11,418-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler2) [294ed6ab] VM '5885c43a-b133-45dc-b8d5-1aa2a4a19579'(prs) moved from 'NotResponding' --> 'Up' 2017-10-01 15:32:11,418-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler2) [294ed6ab] VM 'ae65a104-ed3f-40e0-ba92-5708cf338918'(vpn2) moved from 'NotResponding' --> 'Up' 2017-10-01 15:32:11,418-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler2) [294ed6ab] VM 'd56363ad-9cb1-4a70-a33a-ce439b044d7d'(svn) moved from 'NotResponding' --> 'Up' 2017-10-01 15:32:11,426-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVDSCommand] (DefaultQuartzScheduler2) [294ed6ab] START, FullListVDSCommand(HostName = virt4, FullListVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vmIds='[bcd5776e-978b-4723-933d-70a80322a796]'}), log id: 2986349e 2017-10-01 15:32:11,428-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler4) [27947e4] VM '899e3a1c-dcc4-4426-9dd4-0f1e8b94a2b8'(ipa) moved from 'NotResponding' --> 'Up' 2017-10-01 15:32:11,461-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVDSCommand] (DefaultQuartzScheduler2) [294ed6ab] FINISH, FullListVDSCommand, return: [{username=Unknown, acpiEnable=true, emulatedMachine=pc-i440fx-rhel7.3.0, pid=136180, guestDiskMapping={}, transparentHugePages=true, timeOffset=-14400, cpuType=SandyBridge, smp=2, pauseCode=NOERR, guestNumaNodes=[Ljava.lang.Object;@ee94b23, hypervEnable=true, display=qxl, custom={device_5c6e196f-57c1-4c83-91d9-e9a9e03ed3e2device_f1722274-0033-46fe-b377-0e566f2b0d83device_d15d3fbc-3da0-4b7a-9069-50d68508f35ddevice_2dbbcd0a-697b-4c2e-9524-c46214a299a9device_5d51bbd6-5336-486d-a08c-696efb563560device_75b0e5e0-abec-44d3-ac6e-443e4ca6a264device_cb40e4e4-5329-4f4f-958d-a4269780206d=VmDevice:{id='VmDeviceId:{deviceId='cb40e4e4-5329-4f4f-958d-a4269780206d', vmId='bcd5776e-978b-4723-933d-70a80322a796'}', device='spicevmc', type='REDIRDEV', bootOrder='0', specParams='[]', address='{bus=0, type=usb, port=3}', managed='false', plugged='true', readOnly='false', deviceAlias='redir2', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_5c6e196f-57c1-4c83-91d9-e9a9e03ed3e2device_f1722274-0033-46fe-b377-0e566f2b0d83device_d15d3fbc-3da0-4b7a-9069-50d68508f35ddevice_2dbbcd0a-697b-4c2e-9524-c46214a299a9=VmDevice:{id='VmDeviceId:{deviceId='2dbbcd0a-697b-4c2e-9524-c46214a299a9', vmId='bcd5776e-978b-4723-933d-70a80322a796'}', device='spicevmc', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=3}', managed='false', plugged='true', readOnly='false', deviceAlias='channel2', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_5c6e196f-57c1-4c83-91d9-e9a9e03ed3e2=VmDevice:{id='VmDeviceId:{deviceId='5c6e196f-57c1-4c83-91d9-e9a9e03ed3e2', vmId='bcd5776e-978b-4723-933d-70a80322a796'}', device='ide', type='CONTROLLER', bootOrder='0', specParams='[]', address='{slot=0x01, bus=0x00, domain=0x0000, type=pci, function=0x1}', managed='false', plugged='true', readOnly='false', deviceAlias='ide', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_5c6e196f-57c1-4c83-91d9-e9a9e03ed3e2device_f1722274-0033-46fe-b377-0e566f2b0d83device_d15d3fbc-3da0-4b7a-9069-50d68508f35ddevice_2dbbcd0a-697b-4c2e-9524-c46214a299a9device_5d51bbd6-5336-486d-a08c-696efb563560device_75b0e5e0-abec-44d3-ac6e-443e4ca6a264device_cb40e4e4-5329-4f4f-958d-a4269780206ddevice_de3604de-a806-4a14-ab90-ace38e483fa2=VmDevice:{id='VmDeviceId:{deviceId='de3604de-a806-4a14-ab90-ace38e483fa2', vmId='bcd5776e-978b-4723-933d-70a80322a796'}', device='spicevmc', type='REDIRDEV', bootOrder='0', specParams='[]', address='{bus=0, type=usb, port=4}', managed='false', plugged='true', readOnly='false', deviceAlias='redir3', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_5c6e196f-57c1-4c83-91d9-e9a9e03ed3e2device_f1722274-0033-46fe-b377-0e566f2b0d83device_d15d3fbc-3da0-4b7a-9069-50d68508f35ddevice_2dbbcd0a-697b-4c2e-9524-c46214a299a9device_5d51bbd6-5336-486d-a08c-696efb563560=VmDevice:{id='VmDeviceId:{deviceId='5d51bbd6-5336-486d-a08c-696efb563560', vmId='bcd5776e-978b-4723-933d-70a80322a796'}', device='spicevmc', type='REDIRDEV', bootOrder='0', specParams='[]', address='{bus=0, type=usb, port=1}', managed='false', plugged='true', readOnly='false', deviceAlias='redir0', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_5c6e196f-57c1-4c83-91d9-e9a9e03ed3e2device_f1722274-0033-46fe-b377-0e566f2b0d83device_d15d3fbc-3da0-4b7a-9069-50d68508f35ddevice_2dbbcd0a-697b-4c2e-9524-c46214a299a9device_5d51bbd6-5336-486d-a08c-696efb563560device_75b0e5e0-abec-44d3-ac6e-443e4ca6a264=VmDevice:{id='VmDeviceId:{deviceId='75b0e5e0-abec-44d3-ac6e-443e4ca6a264', vmId='bcd5776e-978b-4723-933d-70a80322a796'}', device='spicevmc', type='REDIRDEV', bootOrder='0', specParams='[]', address='{bus=0, type=usb, port=2}', managed='false', plugged='true', readOnly='false', deviceAlias='redir1', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_5c6e196f-57c1-4c83-91d9-e9a9e03ed3e2device_f1722274-0033-46fe-b377-0e566f2b0d83device_d15d3fbc-3da0-4b7a-9069-50d68508f35d=VmDevice:{id='VmDeviceId:{deviceId='d15d3fbc-3da0-4b7a-9069-50d68508f35d', vmId='bcd5776e-978b-4723-933d-70a80322a796'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=2}', managed='false', plugged='true', readOnly='false', deviceAlias='channel1', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_5c6e196f-57c1-4c83-91d9-e9a9e03ed3e2device_f1722274-0033-46fe-b377-0e566f2b0d83=VmDevice:{id='VmDeviceId:{deviceId='f1722274-0033-46fe-b377-0e566f2b0d83', vmId='bcd5776e-978b-4723-933d-70a80322a796'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=1}', managed='false', plugged='true', readOnly='false', deviceAlias='channel0', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}}, vmType=kvm, memSize=8192, smpCoresPerSocket=1, vmName=cape, nice=0, guestFQDN=, bootMenuEnable=false, vmId=bcd5776e-978b-4723-933d-70a80322a796, smpThreadsPerCore=1, smartcardEnable=false, guestIPs=, kvmEnable=true, displayNetwork=yorku, devices=[Ljava.lang.Object;@1ac299c1, memGuaranteedSize=5461, status=Up, maxVCpus=16, clientIp=, statusTime=5253636100, maxMemSlots=16}], log id: 2986349e 2017-10-01 15:32:11,465-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler2) [294ed6ab] Received a spicevmc Device without an address when processing VM bcd5776e-978b-4723-933d-70a80322a796 devices, skipping device: {device=spicevmc, bus=usb, type=redir, deviceId=8241e2e6-7340-40c8-a4f1-92e4ccff440c, specParams={}} 2017-10-01 15:32:11,465-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler2) [294ed6ab] Received a spicevmc Device without an address when processing VM bcd5776e-978b-4723-933d-70a80322a796 devices, skipping device: {device=spicevmc, bus=usb, type=redir, deviceId=9a03ca98-987d-4309-bd8b-20fd230c9057, specParams={}} 2017-10-01 15:32:11,465-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler2) [294ed6ab] Received a spicevmc Device without an address when processing VM bcd5776e-978b-4723-933d-70a80322a796 devices, skipping device: {device=spicevmc, bus=usb, type=redir, deviceId=f60084fb-a381-49f3-8006-d45bc17e0312, specParams={}} 2017-10-01 15:32:11,465-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler2) [294ed6ab] Received a spicevmc Device without an address when processing VM bcd5776e-978b-4723-933d-70a80322a796 devices, skipping device: {device=spicevmc, bus=usb, type=redir, deviceId=48522fd4-dfea-48f7-83a4-9c574b884f77, specParams={}} 2017-10-01 15:32:11,465-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler2) [294ed6ab] Received a spice Device without an address when processing VM bcd5776e-978b-4723-933d-70a80322a796 devices, skipping device: {device=spice, specParams={fileTransferEnable=true, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,ssmartcard,susbredir, copyPasteEnable=true, displayIp=130.63.94.37, displayNetwork=yorku}, type=graphics, deviceId=a09ffaa5-fede-473c-a53a-986424dce8e4, tlsPort=5908} 2017-10-01 15:32:11,465-04 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler2) [294ed6ab] VM 'bcd5776e-978b-4723-933d-70a80322a796' managed non pluggable device was removed unexpectedly from libvirt: 'VmDevice:{id='VmDeviceId:{deviceId='48522fd4-dfea-48f7-83a4-9c574b884f77', vmId='bcd5776e-978b-4723-933d-70a80322a796'}', device='spicevmc', type='REDIR', bootOrder='0', specParams='[]', address='', managed='true', plugged='false', readOnly='false', deviceAlias='', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}' 2017-10-01 15:32:11,465-04 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler2) [294ed6ab] VM 'bcd5776e-978b-4723-933d-70a80322a796' managed non pluggable device was removed unexpectedly from libvirt: 'VmDevice:{id='VmDeviceId:{deviceId='8241e2e6-7340-40c8-a4f1-92e4ccff440c', vmId='bcd5776e-978b-4723-933d-70a80322a796'}', device='spicevmc', type='REDIR', bootOrder='0', specParams='[]', address='', managed='true', plugged='false', readOnly='false', deviceAlias='', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}' 2017-10-01 15:32:11,465-04 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler2) [294ed6ab] VM 'bcd5776e-978b-4723-933d-70a80322a796' managed non pluggable device was removed unexpectedly from libvirt: 'VmDevice:{id='VmDeviceId:{deviceId='9a03ca98-987d-4309-bd8b-20fd230c9057', vmId='bcd5776e-978b-4723-933d-70a80322a796'}', device='spicevmc', type='REDIR', bootOrder='0', specParams='[]', address='', managed='true', plugged='false', readOnly='false', deviceAlias='', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}' 2017-10-01 15:32:11,465-04 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler2) [294ed6ab] VM 'bcd5776e-978b-4723-933d-70a80322a796' managed non pluggable device was removed unexpectedly from libvirt: 'VmDevice:{id='VmDeviceId:{deviceId='f60084fb-a381-49f3-8006-d45bc17e0312', vmId='bcd5776e-978b-4723-933d-70a80322a796'}', device='spicevmc', type='REDIR', bootOrder='0', specParams='[]', address='', managed='true', plugged='false', readOnly='false', deviceAlias='', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}' 2017-10-01 15:32:20,454-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler7) [39bd9aba] spmStart polling ended: taskId 'cc096f6f-7c74-4c60-97ba-62914e10f68d' task status 'finished' 2017-10-01 15:32:21,467-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler7) [39bd9aba] spmStart polling ended, spm status: SPM 2017-10-01 15:32:21,467-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler7) [39bd9aba] START, HSMClearTaskVDSCommand(HostName = virt3, HSMTaskGuidBaseVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', taskId='cc096f6f-7c74-4c60-97ba-62914e10f68d'}), log id: 5713b0b1 2017-10-01 15:32:22,477-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler7) [39bd9aba] FINISH, HSMClearTaskVDSCommand, log id: 5713b0b1 2017-10-01 15:32:22,477-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler7) [39bd9aba] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@12817a3c, log id: 5d7f43da 2017-10-01 15:32:22,478-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler7) [39bd9aba] Initialize Irs proxy from vds: virt3-mgmt 2017-10-01 15:32:22,507-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [39bd9aba] EVENT_ID: IRS_HOSTED_ON_VDS(204), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host virt3 (Address: virt3-mgmt). 2017-10-01 15:32:22,507-04 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to virt3-mgmt/192.168.0.36 2017-10-01 15:32:22,515-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-6-thread-34) [39bd9aba] START, SPMGetAllTasksInfoVDSCommand( IrsBaseVDSCommandParameters:{runAsync='true', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', ignoreFailoverLimit='false'}), log id: 3f24d195 2017-10-01 15:32:24,588-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-6-thread-34) [39bd9aba] -- executeIrsBrokerCommand: Attempting on storage pool 'aca61eac-ed8a-484b-bbee-3645148f6405' 2017-10-01 15:32:24,589-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-6-thread-34) [39bd9aba] START, HSMGetAllTasksInfoVDSCommand(HostName = virt3, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736'}), log id: 4e96ed31 2017-10-01 15:32:25,599-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-6-thread-34) [39bd9aba] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4e96ed31 2017-10-01 15:32:25,599-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-6-thread-34) [39bd9aba] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 3f24d195 2017-10-01 15:32:25,599-04 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-6-thread-34) [39bd9aba] Discovered no tasks on Storage Pool 'EECS-Primary' 2017-10-01 15:32:26,480-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler7) [7398d46e] VM '77766cb4-0625-4432-a16f-def5e702102a'(labtest1) moved from 'NotResponding' --> 'Up' 2017-10-01 15:32:26,553-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler4) [df91006] VM '9ea2821e-557f-45cb-9872-1e6c5aeaf246'(rs) moved from 'NotResponding' --> 'Up' 2017-10-01 15:32:31,544-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler1) [460a1e35] VM 'c64d80f9-a3bd-44f3-89c4-f1b9b509cdde'(red) moved from 'NotResponding' --> 'Up' 2017-10-01 15:32:33,562-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-12) [] Host 'e1e71e9d-a0cd-47db-96c6-f724f6cea736' no longer storage access problem to any relevant domain clearing it's report (report id: 'ecf7ba6a-babd-4772-9dee-e6eba39645ec') 2017-10-01 15:32:33,563-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-12) [] Domain '63d680c7-4c94-4b29-a6cb-f288a128b9bc:iso' recovered from problem. vds: 'virt3' 2017-10-01 15:32:33,563-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-12) [] Domain '63d680c7-4c94-4b29-a6cb-f288a128b9bc:iso' has recovered from problem. No active host in the DC is reporting it as problematic, so clearing the domain recovery timer. 2017-10-01 15:32:36,926-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler6) [d079f4d] No hosts has reported storage access problem to domains, clearing the handled hosts reports map 2017-10-01 15:32:41,542-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler9) [4adcf0d7] VM '59287796-c634-4f3c-94b1-2c9699986637'(vpn1) moved from 'NotResponding' --> 'Up' 2017-10-01 15:32:55,643-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler7) [65063a1] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 15:32:56,595-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler6) [52483a8] VM '07331c78-6e09-41b4-9b29-edda32f46351'(samba) moved from 'NotResponding' --> 'Up' 2017-10-01 15:32:56,595-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler6) [52483a8] VM 'dc66c051-24b7-4cb8-99eb-db398d396ef1'(samba1) moved from 'NotResponding' --> 'Up' 2017-10-01 15:33:55,662-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [447e3d72] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 15:34:07,416-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler9) [46b78806] VM '1734fed5-fc97-4714-bb63-55e8f4eb6d34'(wiki) moved from 'NotResponding' --> 'Paused' 2017-10-01 15:34:07,487-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [46b78806] EVENT_ID: VM_PAUSED(1,025), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM wiki has been paused. 2017-10-01 15:34:07,508-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [46b78806] EVENT_ID: VM_PAUSED_EIO(145), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM wiki has been paused due to storage I/O problem. 2017-10-01 15:34:23,527-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler1) [14cde17f] VM 'cc1dbe93-bb80-451e-8b3b-c9cc5e90432f'(license) moved from 'NotResponding' --> 'Paused' 2017-10-01 15:34:23,601-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [14cde17f] EVENT_ID: VM_PAUSED(1,025), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM license has been paused. 2017-10-01 15:34:23,623-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [14cde17f] EVENT_ID: VM_PAUSED_EIO(145), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM license has been paused due to storage I/O problem. 2017-10-01 15:34:25,294-04 INFO [org.ovirt.engine.core.bll.SetVmTicketCommand] (default task-13) [6cd31ef2] Running command: SetVmTicketCommand internal: false. Entities affected : ID: 1734fed5-fc97-4714-bb63-55e8f4eb6d34 Type: VMAction group CONNECT_TO_VM with role type USER 2017-10-01 15:34:25,324-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (default task-13) [6cd31ef2] START, SetVmTicketVDSCommand(HostName = virt3, SetVmTicketVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vmId='1734fed5-fc97-4714-bb63-55e8f4eb6d34', protocol='SPICE', ticket='jMb9wpuUppKB', validTime='120', userName='admin', userId='fdfc627c-d875-11e0-90f0-83df133b58cc', disconnectAction='LOCK_SCREEN'}), log id: 5b523803 2017-10-01 15:34:26,379-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (default task-13) [6cd31ef2] FINISH, SetVmTicketVDSCommand, log id: 5b523803 2017-10-01 15:34:26,387-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-13) [6cd31ef2] EVENT_ID: VM_SET_TICKET(164), Correlation ID: 6cd31ef2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: User admin@internal initiated console session for VM wiki 2017-10-01 15:34:39,679-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [8d86fb9] EVENT_ID: VM_CONSOLE_CONNECTED(167), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: User admin@internal is connected to VM wiki. 2017-10-01 15:34:39,680-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler1) [8d86fb9] VM '64734fca-4d6c-443c-82d1-abf7cf106601'(webdav) moved from 'NotResponding' --> 'Paused' 2017-10-01 15:34:39,702-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [8d86fb9] EVENT_ID: VM_PAUSED(1,025), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM webdav has been paused. 2017-10-01 15:34:39,724-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [8d86fb9] EVENT_ID: VM_PAUSED_EIO(145), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM webdav has been paused due to storage I/O problem. 2017-10-01 15:34:55,682-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [7919d756] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 15:35:55,705-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [38808d4d] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 15:36:01,898-04 INFO [org.ovirt.engine.core.bll.SetVmTicketCommand] (default task-22) [54a737dc] Running command: SetVmTicketCommand internal: false. Entities affected : ID: 64734fca-4d6c-443c-82d1-abf7cf106601 Type: VMAction group CONNECT_TO_VM with role type USER 2017-10-01 15:36:01,929-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (default task-22) [54a737dc] START, SetVmTicketVDSCommand(HostName = virt3, SetVmTicketVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vmId='64734fca-4d6c-443c-82d1-abf7cf106601', protocol='SPICE', ticket='zMTWTfPfPIab', validTime='120', userName='admin', userId='fdfc627c-d875-11e0-90f0-83df133b58cc', disconnectAction='LOCK_SCREEN'}), log id: 3aa8aff1 2017-10-01 15:36:02,987-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (default task-22) [54a737dc] FINISH, SetVmTicketVDSCommand, log id: 3aa8aff1 2017-10-01 15:36:03,020-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-22) [54a737dc] EVENT_ID: VM_SET_TICKET(164), Correlation ID: 54a737dc, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: User admin@internal initiated console session for VM webdav 2017-10-01 15:36:12,002-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [4c670f25] EVENT_ID: VM_CONSOLE_CONNECTED(167), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: User admin@internal is connected to VM webdav. 2017-10-01 15:36:55,724-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [1ad432ce] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 15:37:55,747-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [37ee79d] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 15:38:55,767-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [4d8ac17f] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 15:39:12,382-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler8) [4b04b103] VM 'c64d80f9-a3bd-44f3-89c4-f1b9b509cdde'(red) moved from 'Up' --> 'NotResponding' 2017-10-01 15:39:12,409-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [4b04b103] EVENT_ID: VM_NOT_RESPONDING(126), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM red is not responding. 2017-10-01 15:39:55,786-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler7) [bd2261a] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 15:40:55,806-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [3df09a2d] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 15:41:29,894-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [77634fd0] EVENT_ID: VM_CONSOLE_DISCONNECTED(168), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: User admin@internal got disconnected from VM wiki. 2017-10-01 15:41:44,959-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [4e461016] EVENT_ID: VM_CONSOLE_DISCONNECTED(168), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: User admin@internal got disconnected from VM webdav. 2017-10-01 15:41:55,826-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [6e5a297a] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 15:42:55,850-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [71da734] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 15:43:55,869-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [25149e0d] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 15:44:15,104-04 ERROR [org.ovirt.engine.core.sso.servlets.InteractiveAuthServlet] (default task-114) [] Cannot authenticate user 'admin@eecs.yorku.ca': The username or password is incorrect. 2017-10-01 15:44:29,709-04 ERROR [org.ovirt.engine.core.sso.servlets.InteractiveAuthServlet] (default task-128) [] Cannot authenticate user 'admin@internal': The username or password is incorrect. 2017-10-01 15:44:41,439-04 INFO [org.ovirt.engine.core.sso.utils.AuthenticationUtils] (default task-136) [] User admin@internal successfully logged in with scopes: ovirt-app-admin ovirt-app-api ovirt-app-portal ovirt-ext=auth:sequence-priority=~ ovirt-ext=revoke:revoke-all ovirt-ext=token-info:authz-search ovirt-ext=token-info:public-authz-search ovirt-ext=token-info:validate ovirt-ext=token:password-access 2017-10-01 15:44:41,514-04 INFO [org.ovirt.engine.core.bll.aaa.CreateUserSessionCommand] (default task-138) [1163d2a1] Running command: CreateUserSessionCommand internal: false. 2017-10-01 15:44:41,523-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-138) [1163d2a1] EVENT_ID: USER_VDC_LOGIN(30), Correlation ID: 1163d2a1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: User admin@internal logged in. 2017-10-01 15:44:51,081-04 ERROR [org.ovirt.engine.core.utils.servlet.ServletUtils] (default task-162) [] Can't read file '/usr/share/ovirt-engine/files/spice/SpiceVersion.txt' for request '/ovirt-engine/services/files/spice/SpiceVersion.txt', will send a 404 error response. 2017-10-01 15:44:55,890-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [10b0b860] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 15:45:55,907-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [29377b02] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 15:46:45,827-04 INFO [org.ovirt.engine.core.bll.StopVmCommand] (org.ovirt.thread.pool-6-thread-5) [b97325fc-b8a8-4c9c-a551-e6938322d853] Running command: StopVmCommand internal: false. Entities affected : ID: 1734fed5-fc97-4714-bb63-55e8f4eb6d34 Type: VMAction group STOP_VM with role type USER 2017-10-01 15:46:45,844-04 INFO [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (org.ovirt.thread.pool-6-thread-5) [b97325fc-b8a8-4c9c-a551-e6938322d853] START, DestroyVmVDSCommand( DestroyVmVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vmId='1734fed5-fc97-4714-bb63-55e8f4eb6d34', force='false', secondsToWait='0', gracefully='false', reason='', ignoreNoVm='false'}), log id: 70e114ab 2017-10-01 15:46:45,845-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-6-thread-5) [b97325fc-b8a8-4c9c-a551-e6938322d853] START, DestroyVDSCommand(HostName = virt3, DestroyVmVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vmId='1734fed5-fc97-4714-bb63-55e8f4eb6d34', force='false', secondsToWait='0', gracefully='false', reason='', ignoreNoVm='false'}), log id: 159933b9 2017-10-01 15:46:46,398-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-6-thread-5) [b97325fc-b8a8-4c9c-a551-e6938322d853] FINISH, DestroyVDSCommand, log id: 159933b9 2017-10-01 15:46:46,398-04 INFO [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (org.ovirt.thread.pool-6-thread-5) [b97325fc-b8a8-4c9c-a551-e6938322d853] FINISH, DestroyVmVDSCommand, log id: 70e114ab 2017-10-01 15:46:46,405-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-5) [b97325fc-b8a8-4c9c-a551-e6938322d853] EVENT_ID: USER_STOP_VM(33), Correlation ID: b97325fc-b8a8-4c9c-a551-e6938322d853, Job ID: e8d3c86c-290a-48aa-b723-65294c763b99, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM wiki powered off by admin@internal (Host: virt3). 2017-10-01 15:46:53,665-04 INFO [org.ovirt.engine.core.bll.StopVmCommand] (org.ovirt.thread.pool-6-thread-25) [8f6b52ee-176d-4916-a67a-d54448f72846] Running command: StopVmCommand internal: false. Entities affected : ID: 64734fca-4d6c-443c-82d1-abf7cf106601 Type: VMAction group STOP_VM with role type USER 2017-10-01 15:46:53,683-04 INFO [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (org.ovirt.thread.pool-6-thread-25) [8f6b52ee-176d-4916-a67a-d54448f72846] START, DestroyVmVDSCommand( DestroyVmVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vmId='64734fca-4d6c-443c-82d1-abf7cf106601', force='false', secondsToWait='0', gracefully='false', reason='', ignoreNoVm='false'}), log id: f4724 2017-10-01 15:46:53,683-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-6-thread-25) [8f6b52ee-176d-4916-a67a-d54448f72846] START, DestroyVDSCommand(HostName = virt3, DestroyVmVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vmId='64734fca-4d6c-443c-82d1-abf7cf106601', force='false', secondsToWait='0', gracefully='false', reason='', ignoreNoVm='false'}), log id: 65e2f92e 2017-10-01 15:46:54,245-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmsStatisticsFetcher] (DefaultQuartzScheduler5) [7dff8f1e] Fetched 3 VMs from VDS 'e1e71e9d-a0cd-47db-96c6-f724f6cea736' 2017-10-01 15:46:54,246-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler5) [7dff8f1e] VM '1734fed5-fc97-4714-bb63-55e8f4eb6d34'(wiki) is running in db and not running on VDS 'e1e71e9d-a0cd-47db-96c6-f724f6cea736'(virt3) 2017-10-01 15:46:54,275-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [7dff8f1e] EVENT_ID: VM_DOWN(61), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM wiki is down. 2017-10-01 15:46:54,279-04 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler5) [7dff8f1e] Failed to invoke scheduled method poll: null 2017-10-01 15:46:54,287-04 INFO [org.ovirt.engine.core.bll.ProcessDownVmCommand] (org.ovirt.thread.pool-6-thread-27) [16861f2c] Running command: ProcessDownVmCommand internal: true. 2017-10-01 15:46:55,245-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-6-thread-25) [8f6b52ee-176d-4916-a67a-d54448f72846] FINISH, DestroyVDSCommand, log id: 65e2f92e 2017-10-01 15:46:55,245-04 INFO [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (org.ovirt.thread.pool-6-thread-25) [8f6b52ee-176d-4916-a67a-d54448f72846] FINISH, DestroyVmVDSCommand, log id: f4724 2017-10-01 15:46:55,251-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-25) [8f6b52ee-176d-4916-a67a-d54448f72846] EVENT_ID: USER_STOP_VM(33), Correlation ID: 8f6b52ee-176d-4916-a67a-d54448f72846, Job ID: ccfe1f5d-a3d0-4108-bfed-90794954a93c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM webdav powered off by admin@internal (Host: virt3). 2017-10-01 15:46:55,925-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [39e02156] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 15:47:08,620-04 INFO [org.ovirt.engine.core.bll.StopVmCommand] (org.ovirt.thread.pool-6-thread-3) [77754383-103e-47ac-80f7-ce14dd6bceb6] Running command: StopVmCommand internal: false. Entities affected : ID: 64734fca-4d6c-443c-82d1-abf7cf106601 Type: VMAction group STOP_VM with role type USER 2017-10-01 15:47:08,636-04 INFO [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (org.ovirt.thread.pool-6-thread-3) [77754383-103e-47ac-80f7-ce14dd6bceb6] START, DestroyVmVDSCommand( DestroyVmVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vmId='64734fca-4d6c-443c-82d1-abf7cf106601', force='false', secondsToWait='0', gracefully='false', reason='', ignoreNoVm='false'}), log id: 7887b6d9 2017-10-01 15:47:08,636-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-6-thread-3) [77754383-103e-47ac-80f7-ce14dd6bceb6] START, DestroyVDSCommand(HostName = virt3, DestroyVmVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vmId='64734fca-4d6c-443c-82d1-abf7cf106601', force='false', secondsToWait='0', gracefully='false', reason='', ignoreNoVm='false'}), log id: 1e7c8069 2017-10-01 15:47:09,285-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-6-thread-3) [77754383-103e-47ac-80f7-ce14dd6bceb6] Destroy VM couldn't find VM '64734fca-4d6c-443c-82d1-abf7cf106601'. If after Cancel Migration and VM is UP on source, ignore next error, it's just a clean-up call 2017-10-01 15:47:09,286-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-6-thread-3) [77754383-103e-47ac-80f7-ce14dd6bceb6] Failed in 'DestroyVDS' method 2017-10-01 15:47:09,291-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-3) [77754383-103e-47ac-80f7-ce14dd6bceb6] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command DestroyVDS failed: Virtual machine does not exist 2017-10-01 15:47:09,291-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-6-thread-3) [77754383-103e-47ac-80f7-ce14dd6bceb6] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand' return value 'StatusOnlyReturn [status=Status [code=1, message=Virtual machine does not exist]]' 2017-10-01 15:47:09,291-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-6-thread-3) [77754383-103e-47ac-80f7-ce14dd6bceb6] HostName = virt3 2017-10-01 15:47:09,291-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-6-thread-3) [77754383-103e-47ac-80f7-ce14dd6bceb6] Command 'DestroyVDSCommand(HostName = virt3, DestroyVmVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vmId='64734fca-4d6c-443c-82d1-abf7cf106601', force='false', secondsToWait='0', gracefully='false', reason='', ignoreNoVm='false'})' execution failed: VDSGenericException: VDSErrorException: Failed to DestroyVDS, error = Virtual machine does not exist, code = 1 2017-10-01 15:47:09,291-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-6-thread-3) [77754383-103e-47ac-80f7-ce14dd6bceb6] FINISH, DestroyVDSCommand, log id: 1e7c8069 2017-10-01 15:47:09,291-04 ERROR [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (org.ovirt.thread.pool-6-thread-3) [77754383-103e-47ac-80f7-ce14dd6bceb6] Failed to destroy VM '64734fca-4d6c-443c-82d1-abf7cf106601' in VDS = 'e1e71e9d-a0cd-47db-96c6-f724f6cea736' , error = 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to DestroyVDS, error = Virtual machine does not exist, code = 1' 2017-10-01 15:47:09,291-04 INFO [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (org.ovirt.thread.pool-6-thread-3) [77754383-103e-47ac-80f7-ce14dd6bceb6] FINISH, DestroyVmVDSCommand, log id: 7887b6d9 2017-10-01 15:47:09,292-04 ERROR [org.ovirt.engine.core.bll.StopVmCommand] (org.ovirt.thread.pool-6-thread-3) [77754383-103e-47ac-80f7-ce14dd6bceb6] Command 'org.ovirt.engine.core.bll.StopVmCommand' failed: EngineException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to DestroyVDS, error = Virtual machine does not exist, code = 1 (Failed with error noVM and code 1) 2017-10-01 15:47:09,303-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-3) [77754383-103e-47ac-80f7-ce14dd6bceb6] EVENT_ID: USER_FAILED_STOP_VM(56), Correlation ID: 77754383-103e-47ac-80f7-ce14dd6bceb6, Job ID: 42f3893e-55c9-47a9-99a0-f1f1090e985f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to power off VM webdav (Host: virt3, User: admin@internal). 2017-10-01 15:47:10,290-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmsStatisticsFetcher] (DefaultQuartzScheduler1) [7bed6169] Fetched 2 VMs from VDS 'e1e71e9d-a0cd-47db-96c6-f724f6cea736' 2017-10-01 15:47:10,290-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler1) [7bed6169] VM '64734fca-4d6c-443c-82d1-abf7cf106601'(webdav) is running in db and not running on VDS 'e1e71e9d-a0cd-47db-96c6-f724f6cea736'(virt3) 2017-10-01 15:47:10,318-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [7bed6169] EVENT_ID: VM_DOWN(61), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM webdav is down. 2017-10-01 15:47:10,321-04 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler1) [7bed6169] Failed to invoke scheduled method poll: null 2017-10-01 15:47:10,375-04 INFO [org.ovirt.engine.core.bll.ProcessDownVmCommand] (org.ovirt.thread.pool-6-thread-22) [3c9cae36] Running command: ProcessDownVmCommand internal: true. 2017-10-01 15:47:55,948-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [51760596] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 15:48:55,968-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [2d74bac4] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 15:49:08,853-04 INFO [org.ovirt.engine.core.bll.RunVmCommand] (default task-358) [e5814c86-0da1-4f8b-900d-ba56e391d502] Lock Acquired to object 'EngineLock:{exclusiveLocks='[64734fca-4d6c-443c-82d1-abf7cf106601=VM]', sharedLocks=''}' 2017-10-01 15:49:08,916-04 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (default task-358) [e5814c86-0da1-4f8b-900d-ba56e391d502] START, IsVmDuringInitiatingVDSCommand( IsVmDuringInitiatingVDSCommandParameters:{runAsync='true', vmId='64734fca-4d6c-443c-82d1-abf7cf106601'}), log id: 1353bf8 2017-10-01 15:49:08,916-04 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (default task-358) [e5814c86-0da1-4f8b-900d-ba56e391d502] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 1353bf8 2017-10-01 15:49:09,023-04 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-6-thread-36) [e5814c86-0da1-4f8b-900d-ba56e391d502] Running command: RunVmCommand internal: false. Entities affected : ID: 64734fca-4d6c-443c-82d1-abf7cf106601 Type: VMAction group RUN_VM with role type USER 2017-10-01 15:49:09,131-04 INFO [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (org.ovirt.thread.pool-6-thread-36) [e5814c86-0da1-4f8b-900d-ba56e391d502] START, UpdateVmDynamicDataVDSCommand( UpdateVmDynamicDataVDSCommandParameters:{runAsync='true', hostId='null', vmId='64734fca-4d6c-443c-82d1-abf7cf106601', vmDynamic='org.ovirt.engine.core.common.businessentities.VmDynamic@8436478'}), log id: 547336f7 2017-10-01 15:49:09,133-04 INFO [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (org.ovirt.thread.pool-6-thread-36) [e5814c86-0da1-4f8b-900d-ba56e391d502] FINISH, UpdateVmDynamicDataVDSCommand, log id: 547336f7 2017-10-01 15:49:09,136-04 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-6-thread-36) [e5814c86-0da1-4f8b-900d-ba56e391d502] START, CreateVmVDSCommand( CreateVmVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vmId='64734fca-4d6c-443c-82d1-abf7cf106601', vm='VM [webdav]'}), log id: 5bbf63b4 2017-10-01 15:49:09,138-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-6-thread-36) [e5814c86-0da1-4f8b-900d-ba56e391d502] START, CreateVDSCommand(HostName = virt1, CreateVmVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vmId='64734fca-4d6c-443c-82d1-abf7cf106601', vm='VM [webdav]'}), log id: 26c3ef7d 2017-10-01 15:49:09,141-04 INFO [org.ovirt.engine.core.vdsbroker.builder.vminfo.VmInfoBuilderImpl] (org.ovirt.thread.pool-6-thread-36) [e5814c86-0da1-4f8b-900d-ba56e391d502] Bootable disk 'ec77b6c9-87e6-4886-979a-945d05767d03' set to index '0' 2017-10-01 15:49:09,158-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-6-thread-36) [e5814c86-0da1-4f8b-900d-ba56e391d502] VM {memGuaranteedSize=682, smpThreadsPerCore=1, cpuType=SandyBridge, vmId=64734fca-4d6c-443c-82d1-abf7cf106601, acpiEnable=true, vmType=kvm, smp=1, smpCoresPerSocket=1, emulatedMachine=pc-i440fx-rhel7.3.0, smartcardEnable=false, guestNumaNodes=[{memory=1024, cpus=0, nodeIndex=0}], transparentHugePages=true, displayNetwork=yorku, vmName=webdav, maxVCpus=16, kvmEnable=true, devices=[{address={bus=0x00, domain=0x0000, function=0x0, slot=0x02, type=pci}, type=video, specParams={heads=1, vram=32768}, device=qxl, deviceId=ee3d0719-91d1-41ab-9552-4b3ca4aa427a}, {type=graphics, specParams={fileTransferEnable=true, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,ssmartcard,susbredir, copyPasteEnable=true}, device=spice, deviceId=00000021-0021-0021-0021-00000000022e}, {iface=ide, shared=false, path=, address={bus=1, controller=0, unit=0, type=drive, target=0}, readonly=true, index=2, type=disk, specParams={path=}, device=cdrom, deviceId=254c4475-ea71-4e6b-9c95-4acb162cd3f0}, {discard=false, shared=false, address={bus=0x00, domain=0x0000, function=0x0, slot=0x07, type=pci}, imageID=ec77b6c9-87e6-4886-979a-945d05767d03, format=raw, index=0, optional=false, type=disk, deviceId=ec77b6c9-87e6-4886-979a-945d05767d03, domainID=c8653dba-63ec-4447-872c-0ffcc6af9c0f, propagateErrors=off, iface=virtio, readonly=false, bootOrder=1, poolID=aca61eac-ed8a-484b-bbee-3645148f6405, volumeID=7ac5f512-da36-40c0-a798-57369280dfbe, specParams={}, device=disk}, {discard=false, shared=false, address={bus=0x00, domain=0x0000, function=0x0, slot=0x08, type=pci}, imageID=00d67eef-b095-4db4-a2bd-7ac3db618a36, format=raw, optional=false, type=disk, deviceId=00d67eef-b095-4db4-a2bd-7ac3db618a36, domainID=c8653dba-63ec-4447-872c-0ffcc6af9c0f, propagateErrors=off, iface=virtio, readonly=false, poolID=aca61eac-ed8a-484b-bbee-3645148f6405, volumeID=d63967cb-0fc1-4856-b05b-38f034fbee4a, specParams={}, device=disk}, {filter=vdsm-no-mac-spoofing, nicModel=pv, address={bus=0x00, domain=0x0000, function=0x0, slot=0x03, type=pci}, type=interface, specParams={inbound={}, outbound={}}, device=bridge, linkActive=true, deviceId=05efc9be-02ee-4789-82e9-6dc628b88805, macAddr=00:1a:4a:32:3d:23, network=yorku}, {filter=vdsm-no-mac-spoofing, nicModel=pv, address={bus=0x00, domain=0x0000, function=0x0, slot=0x04, type=pci}, type=interface, specParams={inbound={}, outbound={}}, device=bridge, linkActive=true, deviceId=1df93294-942e-4b18-bda9-b569a13982e2, macAddr=00:1a:4a:32:3d:24, network=mrpriv}, {address={bus=0x00, domain=0x0000, function=0x2, slot=0x01, type=pci}, index=0, model=piix3-uhci, type=controller, specParams={}, device=usb, deviceId=a9c7d04c-b190-4569-942a-0dc43767191c}, {address={bus=0x00, function=0x0, domain=0x0000, slot=0x09, type=pci}, type=balloon, specParams={model=virtio}, device=memballoon, deviceId=cb9f42c3-d97c-440f-a7c3-91bd2e910cd5}, {address={bus=0x00, domain=0x0000, function=0x0, slot=0x05, type=pci}, index=0, model=virtio-scsi, type=controller, specParams={}, device=scsi, deviceId=2b01df54-45a6-4a89-82d1-c0438c4751b7}, {address={bus=0x00, domain=0x0000, function=0x0, slot=0x06, type=pci}, type=controller, specParams={}, device=virtio-serial, deviceId=32999d33-d6b1-4d18-a899-b8cbd376b63e}, {model=virtio, type=rng, specParams={source=urandom}, device=virtio, deviceId=6ebf73e6-42da-46d0-9bdd-b621c53d8150}], custom={device_7dbed212-f29d-499e-aaff-bbfe7e3e2b4adevice_4d533b54-910c-438a-8f7b-cc114161b99f=VmDevice:{id='VmDeviceId:{deviceId='4d533b54-910c-438a-8f7b-cc114161b99f', vmId='64734fca-4d6c-443c-82d1-abf7cf106601'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=2}', managed='false', plugged='true', readOnly='false', deviceAlias='channel1', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_7dbed212-f29d-499e-aaff-bbfe7e3e2b4adevice_4d533b54-910c-438a-8f7b-cc114161b99fdevice_88b1f00b-c532-47f6-ac67-78f187bd9f1d=VmDevice:{id='VmDeviceId:{deviceId='88b1f00b-c532-47f6-ac67-78f187bd9f1d', vmId='64734fca-4d6c-443c-82d1-abf7cf106601'}', device='spicevmc', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=3}', managed='false', plugged='true', readOnly='false', deviceAlias='channel2', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_7dbed212-f29d-499e-aaff-bbfe7e3e2b4adevice_4d533b54-910c-438a-8f7b-cc114161b99fdevice_88b1f00b-c532-47f6-ac67-78f187bd9f1ddevice_c9476b47-1659-434c-9758-534cc2a9d88a=VmDevice:{id='VmDeviceId:{deviceId='c9476b47-1659-434c-9758-534cc2a9d88a', vmId='64734fca-4d6c-443c-82d1-abf7cf106601'}', device='ide', type='CONTROLLER', bootOrder='0', specParams='[]', address='{slot=0x01, bus=0x00, domain=0x0000, type=pci, function=0x1}', managed='false', plugged='true', readOnly='false', deviceAlias='ide', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_7dbed212-f29d-499e-aaff-bbfe7e3e2b4a=VmDevice:{id='VmDeviceId:{deviceId='7dbed212-f29d-499e-aaff-bbfe7e3e2b4a', vmId='64734fca-4d6c-443c-82d1-abf7cf106601'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=1}', managed='false', plugged='true', readOnly='false', deviceAlias='channel0', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}}, display=qxl, timeOffset=0, nice=0, maxMemSize=4096, maxMemSlots=16, bootMenuEnable=false, memSize=1024} 2017-10-01 15:49:09,879-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-6-thread-36) [e5814c86-0da1-4f8b-900d-ba56e391d502] FINISH, CreateVDSCommand, log id: 26c3ef7d 2017-10-01 15:49:09,882-04 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-6-thread-36) [e5814c86-0da1-4f8b-900d-ba56e391d502] FINISH, CreateVmVDSCommand, return: WaitForLaunch, log id: 5bbf63b4 2017-10-01 15:49:09,884-04 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-6-thread-36) [e5814c86-0da1-4f8b-900d-ba56e391d502] Lock freed to object 'EngineLock:{exclusiveLocks='[64734fca-4d6c-443c-82d1-abf7cf106601=VM]', sharedLocks=''}' 2017-10-01 15:49:09,895-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-36) [e5814c86-0da1-4f8b-900d-ba56e391d502] EVENT_ID: USER_STARTED_VM(153), Correlation ID: e5814c86-0da1-4f8b-900d-ba56e391d502, Job ID: ced7c978-df70-4064-abaf-dba20b7defe4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM webdav was started by admin@internal (Host: virt1). 2017-10-01 15:49:13,632-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler1) [79e4b941] VM 'c64d80f9-a3bd-44f3-89c4-f1b9b509cdde'(red) moved from 'NotResponding' --> 'Up' 2017-10-01 15:49:13,656-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmsStatisticsFetcher] (DefaultQuartzScheduler6) [23382913] Fetched 4 VMs from VDS '45f8b331-842e-48e7-9df8-56adddb93836' 2017-10-01 15:49:13,657-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler6) [23382913] VM '64734fca-4d6c-443c-82d1-abf7cf106601'(webdav) moved from 'WaitForLaunch' --> 'PoweringUp' 2017-10-01 15:49:13,664-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVDSCommand] (DefaultQuartzScheduler6) [23382913] START, FullListVDSCommand(HostName = virt1, FullListVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vmIds='[64734fca-4d6c-443c-82d1-abf7cf106601]'}), log id: 70550b0f 2017-10-01 15:49:13,678-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVDSCommand] (DefaultQuartzScheduler6) [23382913] FINISH, FullListVDSCommand, return: [{acpiEnable=true, emulatedMachine=pc-i440fx-rhel7.3.0, vmId=64734fca-4d6c-443c-82d1-abf7cf106601, guestDiskMapping={}, transparentHugePages=true, timeOffset=0, cpuType=SandyBridge, smp=1, pauseCode=NOERR, guestNumaNodes=[Ljava.lang.Object;@20d868dd, smartcardEnable=false, custom={device_7dbed212-f29d-499e-aaff-bbfe7e3e2b4adevice_4d533b54-910c-438a-8f7b-cc114161b99fdevice_88b1f00b-c532-47f6-ac67-78f187bd9f1ddevice_c9476b47-1659-434c-9758-534cc2a9d88a=VmDevice:{id='VmDeviceId:{deviceId='c9476b47-1659-434c-9758-534cc2a9d88a', vmId='64734fca-4d6c-443c-82d1-abf7cf106601'}', device='ide', type='CONTROLLER', bootOrder='0', specParams='[]', address='{slot=0x01, bus=0x00, domain=0x0000, type=pci, function=0x1}', managed='false', plugged='true', readOnly='false', deviceAlias='ide', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_7dbed212-f29d-499e-aaff-bbfe7e3e2b4adevice_4d533b54-910c-438a-8f7b-cc114161b99fdevice_88b1f00b-c532-47f6-ac67-78f187bd9f1d=VmDevice:{id='VmDeviceId:{deviceId='88b1f00b-c532-47f6-ac67-78f187bd9f1d', vmId='64734fca-4d6c-443c-82d1-abf7cf106601'}', device='spicevmc', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=3}', managed='false', plugged='true', readOnly='false', deviceAlias='channel2', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_7dbed212-f29d-499e-aaff-bbfe7e3e2b4a=VmDevice:{id='VmDeviceId:{deviceId='7dbed212-f29d-499e-aaff-bbfe7e3e2b4a', vmId='64734fca-4d6c-443c-82d1-abf7cf106601'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=1}', managed='false', plugged='true', readOnly='false', deviceAlias='channel0', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_7dbed212-f29d-499e-aaff-bbfe7e3e2b4adevice_4d533b54-910c-438a-8f7b-cc114161b99f=VmDevice:{id='VmDeviceId:{deviceId='4d533b54-910c-438a-8f7b-cc114161b99f', vmId='64734fca-4d6c-443c-82d1-abf7cf106601'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=2}', managed='false', plugged='true', readOnly='false', deviceAlias='channel1', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}}, vmType=kvm, memSize=1024, smpCoresPerSocket=1, vmName=webdav, nice=0, status=Up, maxMemSize=4096, bootMenuEnable=false, pid=43226, smpThreadsPerCore=1, memGuaranteedSize=682, kvmEnable=true, displayNetwork=yorku, devices=[Ljava.lang.Object;@1e0e9dd1, display=qxl, maxVCpus=16, clientIp=, statusTime=4665344720, maxMemSlots=16}], log id: 70550b0f 2017-10-01 15:49:13,681-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler6) [23382913] Received a spice Device without an address when processing VM 64734fca-4d6c-443c-82d1-abf7cf106601 devices, skipping device: {device=spice, specParams={fileTransferEnable=true, displayNetwork=yorku, copyPasteEnable=true, displayIp=130.63.94.34, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,ssmartcard,susbredir}, type=graphics, deviceId=00000021-0021-0021-0021-00000000022e, tlsPort=5900} 2017-10-01 15:49:55,987-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler7) [2726ab9d] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 15:49:58,798-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler4) [315972d2] VM 'c64d80f9-a3bd-44f3-89c4-f1b9b509cdde'(red) moved from 'Up' --> 'NotResponding' 2017-10-01 15:49:58,869-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [315972d2] EVENT_ID: VM_NOT_RESPONDING(126), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM red is not responding. 2017-10-01 15:50:15,897-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler7) [4d2953f] VM '64734fca-4d6c-443c-82d1-abf7cf106601'(webdav) moved from 'PoweringUp' --> 'Up' 2017-10-01 15:50:15,905-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [4d2953f] EVENT_ID: USER_RUN_VM(32), Correlation ID: e5814c86-0da1-4f8b-900d-ba56e391d502, Job ID: ced7c978-df70-4064-abaf-dba20b7defe4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM webdav started on Host virt1 2017-10-01 15:50:56,010-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [72cb759b] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 15:51:13,646-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-6) [] VM 'ae56c31c-71f7-433a-b198-a1c2c846325e'(forum) moved from 'Up' --> 'Paused' 2017-10-01 15:51:13,717-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ForkJoinPool-1-worker-6) [] EVENT_ID: VM_PAUSED(1,025), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM forum has been paused. 2017-10-01 15:51:13,739-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ForkJoinPool-1-worker-6) [] EVENT_ID: VM_PAUSED_ERROR(139), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM forum has been paused due to unknown storage error. 2017-10-01 15:51:16,040-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler6) [2f3aa58c] VM 'bfe9a524-2238-4bab-a91f-5dc6ed270896'(pcmode) moved from 'Up' --> 'Paused' 2017-10-01 15:51:16,068-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [2f3aa58c] EVENT_ID: VM_PAUSED(1,025), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM pcmode has been paused. 2017-10-01 15:51:16,089-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [2f3aa58c] EVENT_ID: VM_PAUSED_ERROR(139), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM pcmode has been paused due to unknown storage error. 2017-10-01 15:51:16,089-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler6) [2f3aa58c] VM '1325cebf-834e-4571-a3e1-82beb0021ffe'(dns1) moved from 'Up' --> 'Paused' 2017-10-01 15:51:16,110-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [2f3aa58c] EVENT_ID: VM_PAUSED(1,025), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM dns1 has been paused. 2017-10-01 15:51:16,130-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [2f3aa58c] EVENT_ID: VM_PAUSED_ERROR(139), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM dns1 has been paused due to unknown storage error. 2017-10-01 15:51:16,132-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler6) [2f3aa58c] VM 'a8d4e149-55ec-4961-a414-d7d1e1b05a59'(fix) moved from 'Up' --> 'Paused' 2017-10-01 15:51:16,154-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [2f3aa58c] EVENT_ID: VM_PAUSED(1,025), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM fix has been paused. 2017-10-01 15:51:16,175-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [2f3aa58c] EVENT_ID: VM_PAUSED_ERROR(139), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM fix has been paused due to unknown storage error. 2017-10-01 15:51:16,175-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler6) [2f3aa58c] VM 'aa487207-7ff4-465a-9d9b-2a103d50dc77'(ftp1) moved from 'Up' --> 'Paused' 2017-10-01 15:51:16,196-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [2f3aa58c] EVENT_ID: VM_PAUSED(1,025), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM ftp1 has been paused. 2017-10-01 15:51:16,216-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [2f3aa58c] EVENT_ID: VM_PAUSED_ERROR(139), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM ftp1 has been paused due to unknown storage error. 2017-10-01 15:51:23,323-04 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-5) [] domain 'f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2:export' in problem 'PROBLEMATIC'. vds: 'virt1' 2017-10-01 15:51:23,324-04 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-5) [] domain 'c8653dba-63ec-4447-872c-0ffcc6af9c0f:data' in problem 'PROBLEMATIC'. vds: 'virt1' 2017-10-01 15:51:23,324-04 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-5) [] domain '63d680c7-4c94-4b29-a6cb-f288a128b9bc:iso' in problem 'PROBLEMATIC'. vds: 'virt1' 2017-10-01 15:51:24,809-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler1) [18310084] Failed in 'SpmStatusVDS' method 2017-10-01 15:51:24,813-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [18310084] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command SpmStatusVDS failed: (116, 'Sanlock resource read failure', 'Stale file handle') 2017-10-01 15:51:24,813-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler1) [18310084] Command 'SpmStatusVDSCommand(HostName = virt3, SpmStatusVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: VDSGenericException: VDSErrorException: Failed to SpmStatusVDS, error = (116, 'Sanlock resource read failure', 'Stale file handle'), code = 100 2017-10-01 15:51:24,818-04 INFO [org.ovirt.engine.core.bll.storage.pool.SetStoragePoolStatusCommand] (DefaultQuartzScheduler1) [5474f74c] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: aca61eac-ed8a-484b-bbee-3645148f6405 Type: StoragePool 2017-10-01 15:51:24,819-04 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler1) [5474f74c] Storage Pool 'aca61eac-ed8a-484b-bbee-3645148f6405' - Updating Storage Domain '63d680c7-4c94-4b29-a6cb-f288a128b9bc' status from 'Active' to 'Unknown', reason: null 2017-10-01 15:51:24,820-04 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler1) [5474f74c] Storage Pool 'aca61eac-ed8a-484b-bbee-3645148f6405' - Updating Storage Domain 'f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2' status from 'Active' to 'Unknown', reason: null 2017-10-01 15:51:24,820-04 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler1) [5474f74c] Storage Pool 'aca61eac-ed8a-484b-bbee-3645148f6405' - Updating Storage Domain 'c8653dba-63ec-4447-872c-0ffcc6af9c0f' status from 'Active' to 'Unknown', reason: null 2017-10-01 15:51:24,842-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [5474f74c] EVENT_ID: SYSTEM_CHANGE_STORAGE_POOL_STATUS_PROBLEMATIC_WITH_ERROR(987), Correlation ID: 5474f74c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Invalid status on Data Center EECS-Primary. Setting Data Center status to Non Responsive (On host virt3, Error: General Exception). 2017-10-01 15:51:25,373-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [5474f74c] START, ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 56145fd1 2017-10-01 15:51:25,373-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [5474f74c] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 15:51:28,963-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler7) [d2064bd] VM '07331c78-6e09-41b4-9b29-edda32f46351'(samba) moved from 'Up' --> 'Paused' 2017-10-01 15:51:28,984-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [d2064bd] EVENT_ID: VM_PAUSED(1,025), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM samba has been paused. 2017-10-01 15:51:29,005-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [d2064bd] EVENT_ID: VM_PAUSED_ERROR(139), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM samba has been paused due to unknown storage error. 2017-10-01 15:51:29,005-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler7) [d2064bd] VM 'dc66c051-24b7-4cb8-99eb-db398d396ef1'(samba1) moved from 'Up' --> 'Paused' 2017-10-01 15:51:29,026-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [d2064bd] EVENT_ID: VM_PAUSED(1,025), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM samba1 has been paused. 2017-10-01 15:51:29,048-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [d2064bd] EVENT_ID: VM_PAUSED_ERROR(139), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM samba1 has been paused due to unknown storage error. 2017-10-01 15:51:29,048-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler7) [d2064bd] VM '3e5a5728-8940-4fe7-b4af-11fb8e12f116'(www) moved from 'Up' --> 'Paused' 2017-10-01 15:51:29,069-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [d2064bd] EVENT_ID: VM_PAUSED(1,025), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM www has been paused. 2017-10-01 15:51:29,090-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [d2064bd] EVENT_ID: VM_PAUSED_ERROR(139), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM www has been paused due to unknown storage error. 2017-10-01 15:51:29,090-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler7) [d2064bd] VM 'f87ec08d-0fdf-42ba-8aac-c66c404d7aae'(webapp) moved from 'Up' --> 'Paused' 2017-10-01 15:51:29,111-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [d2064bd] EVENT_ID: VM_PAUSED(1,025), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM webapp has been paused. 2017-10-01 15:51:29,132-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [d2064bd] EVENT_ID: VM_PAUSED_ERROR(139), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM webapp has been paused due to unknown storage error. 2017-10-01 15:51:29,132-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler7) [d2064bd] VM '8391f9a7-7fe7-4025-8fba-6273ae587a52'(archive) moved from 'Up' --> 'Paused' 2017-10-01 15:51:29,153-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [d2064bd] EVENT_ID: VM_PAUSED(1,025), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM archive has been paused. 2017-10-01 15:51:29,173-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [d2064bd] EVENT_ID: VM_PAUSED_ERROR(139), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM archive has been paused due to unknown storage error. 2017-10-01 15:51:29,174-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler7) [d2064bd] VM '9ea2821e-557f-45cb-9872-1e6c5aeaf246'(rs) moved from 'Up' --> 'Paused' 2017-10-01 15:51:29,194-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [d2064bd] EVENT_ID: VM_PAUSED(1,025), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM rs has been paused. 2017-10-01 15:51:29,214-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [d2064bd] EVENT_ID: VM_PAUSED_ERROR(139), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM rs has been paused due to unknown storage error. 2017-10-01 15:51:29,214-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler7) [d2064bd] VM 'd66235b4-7c8d-4da2-a572-82ecc9c53a59'(www-las) moved from 'Up' --> 'Paused' 2017-10-01 15:51:29,235-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [d2064bd] EVENT_ID: VM_PAUSED(1,025), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM www-las has been paused. 2017-10-01 15:51:29,255-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [d2064bd] EVENT_ID: VM_PAUSED_ERROR(139), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM www-las has been paused due to unknown storage error. 2017-10-01 15:51:29,256-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler7) [d2064bd] VM 'ae65a104-ed3f-40e0-ba92-5708cf338918'(vpn2) moved from 'Up' --> 'Paused' 2017-10-01 15:51:29,277-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [d2064bd] EVENT_ID: VM_PAUSED(1,025), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM vpn2 has been paused. 2017-10-01 15:51:29,297-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [d2064bd] EVENT_ID: VM_PAUSED_ERROR(139), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM vpn2 has been paused due to unknown storage error. 2017-10-01 15:51:29,297-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler7) [d2064bd] VM 'd56363ad-9cb1-4a70-a33a-ce439b044d7d'(svn) moved from 'Up' --> 'Paused' 2017-10-01 15:51:29,317-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [d2064bd] EVENT_ID: VM_PAUSED(1,025), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM svn has been paused. 2017-10-01 15:51:29,337-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [d2064bd] EVENT_ID: VM_PAUSED_ERROR(139), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM svn has been paused due to unknown storage error. 2017-10-01 15:51:31,228-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler4) [353e0c3e] VM '899e3a1c-dcc4-4426-9dd4-0f1e8b94a2b8'(ipa) moved from 'Up' --> 'Paused' 2017-10-01 15:51:31,239-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler10) [5ca7aea4] VM 'ddf37076-e795-4c87-a4fb-9efcaae76d72'(dc1) moved from 'Up' --> 'Paused' 2017-10-01 15:51:31,250-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [353e0c3e] EVENT_ID: VM_PAUSED(1,025), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM ipa has been paused. 2017-10-01 15:51:31,265-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [5ca7aea4] EVENT_ID: VM_PAUSED(1,025), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM dc1 has been paused. 2017-10-01 15:51:31,274-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [353e0c3e] EVENT_ID: VM_PAUSED_ERROR(139), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM ipa has been paused due to unknown storage error. 2017-10-01 15:51:31,274-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler4) [353e0c3e] VM '77766cb4-0625-4432-a16f-def5e702102a'(labtest1) moved from 'Up' --> 'Paused' 2017-10-01 15:51:31,286-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [5ca7aea4] EVENT_ID: VM_PAUSED_ERROR(139), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM dc1 has been paused due to unknown storage error. 2017-10-01 15:51:31,287-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler10) [5ca7aea4] VM 'a82b4cc3-6ef6-4a8e-97ef-1d4ea1b7e07c'(indigo) moved from 'Up' --> 'Paused' 2017-10-01 15:51:31,299-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [353e0c3e] EVENT_ID: VM_PAUSED(1,025), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM labtest1 has been paused. 2017-10-01 15:51:31,308-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [5ca7aea4] EVENT_ID: VM_PAUSED(1,025), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM indigo has been paused. 2017-10-01 15:51:31,324-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [353e0c3e] EVENT_ID: VM_PAUSED_ERROR(139), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM labtest1 has been paused due to unknown storage error. 2017-10-01 15:51:31,329-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [5ca7aea4] EVENT_ID: VM_PAUSED_ERROR(139), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM indigo has been paused due to unknown storage error. 2017-10-01 15:51:31,330-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler10) [5ca7aea4] VM '4ba80442-b361-4026-ae8c-895fd28042b0'(dist) moved from 'Up' --> 'Paused' 2017-10-01 15:51:31,351-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [5ca7aea4] EVENT_ID: VM_PAUSED(1,025), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM dist has been paused. 2017-10-01 15:51:31,371-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [5ca7aea4] EVENT_ID: VM_PAUSED_ERROR(139), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM dist has been paused due to unknown storage error. 2017-10-01 15:51:50,727-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [5474f74c] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:51:50,727-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [5474f74c] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 15:51:50,727-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [5474f74c] HostName = virt3 2017-10-01 15:51:50,727-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [5474f74c] Command 'ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:51:50,727-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [5474f74c] FINISH, ConnectStoragePoolVDSCommand, log id: 56145fd1 2017-10-01 15:51:50,727-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler1) [5474f74c] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:51:51,264-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [5474f74c] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 15:51:51,264-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler1) [5474f74c] Command 'SpmStatusVDSCommand(HostName = virt3, SpmStatusVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 15:51:51,264-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler1) [5474f74c] Irs placed on server 'e1e71e9d-a0cd-47db-96c6-f724f6cea736' failed. Proceed Failover 2017-10-01 15:51:51,334-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [5474f74c] START, ConnectStoragePoolVDSCommand(HostName = virt1, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vdsId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 15080e2 2017-10-01 15:51:51,334-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [5474f74c] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 15:51:56,030-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [1ec8e2b7] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 15:51:56,360-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler5) [630bbd48] VM 'c64d80f9-a3bd-44f3-89c4-f1b9b509cdde'(red) moved from 'NotResponding' --> 'Paused' 2017-10-01 15:51:56,388-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [630bbd48] EVENT_ID: VM_PAUSED(1,025), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM red has been paused. 2017-10-01 15:51:56,409-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [630bbd48] EVENT_ID: VM_PAUSED_ERROR(139), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM red has been paused due to unknown storage error. 2017-10-01 15:51:59,386-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler9) [ea8d914] VM 'bcd5776e-978b-4723-933d-70a80322a796'(cape) moved from 'Up' --> 'Paused' 2017-10-01 15:51:59,413-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [ea8d914] EVENT_ID: VM_PAUSED(1,025), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM cape has been paused. 2017-10-01 15:51:59,435-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [ea8d914] EVENT_ID: VM_PAUSED_ERROR(139), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM cape has been paused due to unknown storage error. 2017-10-01 15:51:59,435-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler9) [ea8d914] VM '911d6883-607c-464e-b46b-8e6141bdd2de'(hdb) moved from 'Up' --> 'Paused' 2017-10-01 15:51:59,456-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [ea8d914] EVENT_ID: VM_PAUSED(1,025), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM hdb has been paused. 2017-10-01 15:51:59,525-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [ea8d914] EVENT_ID: VM_PAUSED_ERROR(139), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM hdb has been paused due to unknown storage error. 2017-10-01 15:51:59,526-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler9) [ea8d914] VM '1c7e966f-29e8-4788-b64c-86e52f415061'(udb) moved from 'Up' --> 'Paused' 2017-10-01 15:51:59,548-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [ea8d914] EVENT_ID: VM_PAUSED(1,025), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM udb has been paused. 2017-10-01 15:51:59,568-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [ea8d914] EVENT_ID: VM_PAUSED_ERROR(139), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM udb has been paused due to unknown storage error. 2017-10-01 15:52:01,408-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler8) [4e2385b7] VM '59287796-c634-4f3c-94b1-2c9699986637'(vpn1) moved from 'Up' --> 'Paused' 2017-10-01 15:52:01,418-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler9) [6360f42c] VM 'db3e08fb-c111-442d-aeec-1507da30f486'(print) moved from 'Up' --> 'Paused' 2017-10-01 15:52:01,429-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [4e2385b7] EVENT_ID: VM_PAUSED(1,025), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM vpn1 has been paused. 2017-10-01 15:52:01,439-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [6360f42c] EVENT_ID: VM_PAUSED(1,025), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM print has been paused. 2017-10-01 15:52:01,449-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [4e2385b7] EVENT_ID: VM_PAUSED_ERROR(139), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM vpn1 has been paused due to unknown storage error. 2017-10-01 15:52:01,449-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler8) [4e2385b7] VM '64734fca-4d6c-443c-82d1-abf7cf106601'(webdav) moved from 'Up' --> 'Paused' 2017-10-01 15:52:01,464-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [6360f42c] EVENT_ID: VM_PAUSED_ERROR(139), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM print has been paused due to unknown storage error. 2017-10-01 15:52:01,466-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler9) [6360f42c] VM 'c8682859-9e55-460a-9ef0-d03e4a85f50e'(dhcp1) moved from 'Up' --> 'Paused' 2017-10-01 15:52:01,474-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [4e2385b7] EVENT_ID: VM_PAUSED(1,025), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM webdav has been paused. 2017-10-01 15:52:01,489-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [6360f42c] EVENT_ID: VM_PAUSED(1,025), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM dhcp1 has been paused. 2017-10-01 15:52:01,499-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [4e2385b7] EVENT_ID: VM_PAUSED_ERROR(139), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM webdav has been paused due to unknown storage error. 2017-10-01 15:52:01,516-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [6360f42c] EVENT_ID: VM_PAUSED_ERROR(139), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM dhcp1 has been paused due to unknown storage error. 2017-10-01 15:52:01,516-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler9) [6360f42c] VM '3bb83ad5-dc00-445d-8708-2db18c53e0e6'(www-vhost) moved from 'Up' --> 'Paused' 2017-10-01 15:52:01,537-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [6360f42c] EVENT_ID: VM_PAUSED(1,025), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM www-vhost has been paused. 2017-10-01 15:52:01,558-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [6360f42c] EVENT_ID: VM_PAUSED_ERROR(139), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM www-vhost has been paused due to unknown storage error. 2017-10-01 15:52:04,473-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [5474f74c] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt1 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:52:04,474-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [5474f74c] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 15:52:04,474-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [5474f74c] HostName = virt1 2017-10-01 15:52:04,474-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [5474f74c] Command 'ConnectStoragePoolVDSCommand(HostName = virt1, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vdsId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:52:04,474-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [5474f74c] FINISH, ConnectStoragePoolVDSCommand, log id: 15080e2 2017-10-01 15:52:04,474-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler1) [5474f74c] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:52:05,608-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [5474f74c] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt1 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 15:52:05,608-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler1) [5474f74c] Command 'SpmStatusVDSCommand(HostName = virt1, SpmStatusVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 15:52:05,608-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler1) [5474f74c] Irs placed on server '45f8b331-842e-48e7-9df8-56adddb93836' failed. Proceed Failover 2017-10-01 15:52:05,678-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [5474f74c] START, ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: ff7d70d 2017-10-01 15:52:05,678-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [5474f74c] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 15:52:18,792-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [5474f74c] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:52:18,792-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [5474f74c] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 15:52:18,792-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [5474f74c] HostName = virt4 2017-10-01 15:52:18,792-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [5474f74c] Command 'ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:52:18,792-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [5474f74c] FINISH, ConnectStoragePoolVDSCommand, log id: ff7d70d 2017-10-01 15:52:18,792-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler1) [5474f74c] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:52:19,918-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [5474f74c] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 15:52:19,918-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler1) [5474f74c] Command 'SpmStatusVDSCommand(HostName = virt4, SpmStatusVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 15:52:19,918-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler1) [5474f74c] Irs placed on server '5ae29ed2-9c0f-4754-8df3-b43d14726846' failed. Proceed Failover 2017-10-01 15:52:19,976-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [5474f74c] START, ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 475e577e 2017-10-01 15:52:19,976-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [5474f74c] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 15:52:29,893-04 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.Reactor] (SSL Stomp Reactor) [] Unable to process messages Connection reset by peer 2017-10-01 15:52:29,942-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler7) [30559a1c] Command 'GetAllVmStatsVDSCommand(HostName = virt3, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736'})' execution failed: VDSGenericException: VDSNetworkException: Connection reset by peer 2017-10-01 15:52:29,946-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.PollVmStatsRefresher] (DefaultQuartzScheduler7) [30559a1c] Failed to fetch vms info for host 'virt3' - skipping VMs monitoring. 2017-10-01 15:52:29,950-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [258a3223] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command GetStatsVDS failed: Connection reset by peer 2017-10-01 15:52:29,950-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStatsVDSCommand] (DefaultQuartzScheduler5) [258a3223] Command 'GetStatsVDSCommand(HostName = virt3, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vds='Host[virt3,e1e71e9d-a0cd-47db-96c6-f724f6cea736]'})' execution failed: VDSGenericException: VDSNetworkException: Connection reset by peer 2017-10-01 15:52:29,950-04 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [258a3223] Failed getting vds stats, host='virt3'(e1e71e9d-a0cd-47db-96c6-f724f6cea736): org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Connection reset by peer 2017-10-01 15:52:29,950-04 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [258a3223] Failure to refresh host 'virt3' runtime info: VDSGenericException: VDSNetworkException: Connection reset by peer 2017-10-01 15:52:29,950-04 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler5) [258a3223] Failed to refresh VDS, network error, continuing, vds='virt3'(e1e71e9d-a0cd-47db-96c6-f724f6cea736): VDSGenericException: VDSNetworkException: Connection reset by peer 2017-10-01 15:52:29,950-04 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (org.ovirt.thread.pool-6-thread-9) [258a3223] Host 'virt3' is not responding. It will stay in Connecting state for a grace period of 81 seconds and after that an attempt to fence the host will be issued. 2017-10-01 15:52:29,950-04 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (org.ovirt.thread.pool-6-thread-26) [30559a1c] Host 'virt3' is not responding. It will stay in Connecting state for a grace period of 81 seconds and after that an attempt to fence the host will be issued. 2017-10-01 15:52:29,969-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-9) [258a3223] EVENT_ID: VDS_HOST_NOT_RESPONDING_CONNECTING(9,008), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host virt3 is not responding. It will stay in Connecting state for a grace period of 81 seconds and after that an attempt to fence the host will be issued. 2017-10-01 15:52:29,969-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-26) [30559a1c] EVENT_ID: VDS_HOST_NOT_RESPONDING_CONNECTING(9,008), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host virt3 is not responding. It will stay in Connecting state for a grace period of 81 seconds and after that an attempt to fence the host will be issued. 2017-10-01 15:52:32,966-04 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to virt3-mgmt/192.168.0.36 2017-10-01 15:52:32,967-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler9) [567680e9] Command 'GetCapabilitiesVDSCommand(HostName = virt3, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vds='Host[virt3,e1e71e9d-a0cd-47db-96c6-f724f6cea736]'})' execution failed: java.net.ConnectException: Connection refused 2017-10-01 15:52:32,967-04 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [567680e9] Failure to refresh host 'virt3' runtime info: java.net.ConnectException: Connection refused 2017-10-01 15:52:32,967-04 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler9) [567680e9] Failed to refresh VDS, network error, continuing, vds='virt3'(e1e71e9d-a0cd-47db-96c6-f724f6cea736): java.net.ConnectException: Connection refused 2017-10-01 15:52:32,973-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [5474f74c] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:52:32,973-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [5474f74c] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 15:52:32,973-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [5474f74c] HostName = virt2 2017-10-01 15:52:32,973-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [5474f74c] Command 'ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:52:32,973-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [5474f74c] FINISH, ConnectStoragePoolVDSCommand, log id: 475e577e 2017-10-01 15:52:32,973-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler1) [5474f74c] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:52:33,096-04 INFO [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-8) [73ff9f48] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: c8653dba-63ec-4447-872c-0ffcc6af9c0f Type: Storage 2017-10-01 15:52:33,183-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-6-thread-8) [73ff9f48] START, ResetIrsVDSCommand( ResetIrsVDSCommandParameters:{runAsync='true', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', ignoreFailoverLimit='false', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', ignoreStopFailed='true'}), log id: 125054e6 2017-10-01 15:52:33,198-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-6-thread-8) [73ff9f48] START, SpmStopVDSCommand(HostName = virt3, SpmStopVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'}), log id: 5b6725f 2017-10-01 15:52:33,198-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-6-thread-8) [73ff9f48] SpmStopVDSCommand:: vds 'virt3' is in 'Connecting' status - not performing spm stop, pool id 'aca61eac-ed8a-484b-bbee-3645148f6405' 2017-10-01 15:52:33,198-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-6-thread-8) [73ff9f48] FINISH, SpmStopVDSCommand, log id: 5b6725f 2017-10-01 15:52:33,201-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-6-thread-8) [73ff9f48] FINISH, ResetIrsVDSCommand, log id: 125054e6 2017-10-01 15:52:33,201-04 WARN [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-8) [73ff9f48] skipping connect and refresh for all hosts, last master 'true', command status 'true' 2017-10-01 15:52:33,216-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-8) [73ff9f48] EVENT_ID: RECONSTRUCT_MASTER_FAILED(985), Correlation ID: 73ff9f48, Job ID: 12526420-671f-4abf-8fd6-a420c5632357, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center EECS-Primary. 2017-10-01 15:52:33,219-04 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-6-thread-44) [5474f74c] Finished reconstruct for pool 'aca61eac-ed8a-484b-bbee-3645148f6405'. Clearing event queue 2017-10-01 15:52:35,982-04 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to virt3-mgmt/192.168.0.36 2017-10-01 15:52:35,982-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler3) [1947237a] Command 'GetCapabilitiesVDSCommand(HostName = virt3, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vds='Host[virt3,e1e71e9d-a0cd-47db-96c6-f724f6cea736]'})' execution failed: java.net.ConnectException: Connection refused 2017-10-01 15:52:35,983-04 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [1947237a] Failure to refresh host 'virt3' runtime info: java.net.ConnectException: Connection refused 2017-10-01 15:52:35,983-04 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler3) [1947237a] Failed to refresh VDS, network error, continuing, vds='virt3'(e1e71e9d-a0cd-47db-96c6-f724f6cea736): java.net.ConnectException: Connection refused 2017-10-01 15:52:38,999-04 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to virt3-mgmt/192.168.0.36 2017-10-01 15:52:38,999-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler8) [604d0a9a] Command 'GetCapabilitiesVDSCommand(HostName = virt3, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vds='Host[virt3,e1e71e9d-a0cd-47db-96c6-f724f6cea736]'})' execution failed: java.net.ConnectException: Connection refused 2017-10-01 15:52:38,999-04 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [604d0a9a] Failure to refresh host 'virt3' runtime info: java.net.ConnectException: Connection refused 2017-10-01 15:52:38,999-04 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler8) [604d0a9a] Failed to refresh VDS, network error, continuing, vds='virt3'(e1e71e9d-a0cd-47db-96c6-f724f6cea736): java.net.ConnectException: Connection refused 2017-10-01 15:52:42,014-04 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to virt3-mgmt/192.168.0.36 2017-10-01 15:52:42,151-04 ERROR [org.ovirt.vdsm.jsonrpc.client.JsonRpcClient] (ResponseWorker) [] Not able to update response for "99bbac16-8e27-46b5-8253-a8570defa0f7" 2017-10-01 15:52:42,540-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [72ab0a9d] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command GetCapabilitiesVDS failed: Recovering from crash or Initializing 2017-10-01 15:52:42,540-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler10) [72ab0a9d] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturn@52013f35' 2017-10-01 15:52:42,540-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler10) [72ab0a9d] HostName = virt3 2017-10-01 15:52:42,540-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler10) [72ab0a9d] Command 'GetCapabilitiesVDSCommand(HostName = virt3, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vds='Host[virt3,e1e71e9d-a0cd-47db-96c6-f724f6cea736]'})' execution failed: Recovering from crash or Initializing 2017-10-01 15:52:42,559-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [72ab0a9d] EVENT_ID: VDS_INITIALIZING(514), Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSRecoveringException: Recovering from crash or Initializing 2017-10-01 15:52:42,559-04 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler10) [72ab0a9d] Failed to refresh VDS, continuing, vds='virt3'(e1e71e9d-a0cd-47db-96c6-f724f6cea736): Recovering from crash or Initializing 2017-10-01 15:52:43,122-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [7a77ffc9] START, ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 3c63a271 2017-10-01 15:52:43,123-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [7a77ffc9] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 15:52:43,136-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-34) [] Domain 'c8653dba-63ec-4447-872c-0ffcc6af9c0f:data' recovered from problem. vds: 'virt1' 2017-10-01 15:52:44,219-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [7a77ffc9] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:52:44,219-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [7a77ffc9] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 15:52:44,219-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [7a77ffc9] HostName = virt2 2017-10-01 15:52:44,219-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [7a77ffc9] Command 'ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:52:44,219-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [7a77ffc9] FINISH, ConnectStoragePoolVDSCommand, log id: 3c63a271 2017-10-01 15:52:44,219-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler8) [7a77ffc9] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:52:44,636-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [7a77ffc9] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 15:52:44,636-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler8) [7a77ffc9] Command 'SpmStatusVDSCommand(HostName = virt2, SpmStatusVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 15:52:44,636-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler8) [7a77ffc9] Irs placed on server 'a0ccfd11-3402-4eba-a119-4987561b738e' failed. Proceed Failover 2017-10-01 15:52:44,689-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [7a77ffc9] START, ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 4b179bd3 2017-10-01 15:52:44,689-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [7a77ffc9] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 15:52:44,774-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [7a77ffc9] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:52:44,774-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [7a77ffc9] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 15:52:44,774-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [7a77ffc9] HostName = virt4 2017-10-01 15:52:44,774-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [7a77ffc9] Command 'ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:52:44,774-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [7a77ffc9] FINISH, ConnectStoragePoolVDSCommand, log id: 4b179bd3 2017-10-01 15:52:44,774-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler8) [7a77ffc9] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:52:44,775-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-23) [] Domain 'c8653dba-63ec-4447-872c-0ffcc6af9c0f:data' recovered from problem. vds: 'virt2' 2017-10-01 15:52:44,794-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-5) [] Domain 'c8653dba-63ec-4447-872c-0ffcc6af9c0f:data' recovered from problem. vds: 'virt4' 2017-10-01 15:52:44,951-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [7a77ffc9] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 15:52:44,951-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler8) [7a77ffc9] Command 'SpmStatusVDSCommand(HostName = virt4, SpmStatusVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 15:52:44,951-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler8) [7a77ffc9] Irs placed on server '5ae29ed2-9c0f-4754-8df3-b43d14726846' failed. Proceed Failover 2017-10-01 15:52:44,995-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [7a77ffc9] START, ConnectStoragePoolVDSCommand(HostName = virt1, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vdsId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 189e881f 2017-10-01 15:52:44,996-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [7a77ffc9] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 15:52:44,999-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVDSCommand] (DefaultQuartzScheduler3) [6c5c82d6] START, FullListVDSCommand(HostName = virt3, FullListVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vmIds='[c64d80f9-a3bd-44f3-89c4-f1b9b509cdde]'}), log id: 45ae4519 2017-10-01 15:52:45,087-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVDSCommand] (DefaultQuartzScheduler3) [6c5c82d6] FINISH, FullListVDSCommand, return: [{username=Unknown, acpiEnable=true, emulatedMachine=pc-i440fx-rhel7.3.0, afterMigrationStatus=, pid=25406, memGuaranteedSize=16384, transparentHugePages=true, timeOffset=0, cpuType=SandyBridge, smp=6, pauseCode=EOTHER, guestNumaNodes=[Ljava.lang.Object;@7dfb9d72, smartcardEnable=false, custom={device_49fdfc30-a6c4-4ded-8823-2e2a886b1d1c=VmDevice:{id='VmDeviceId:{deviceId='49fdfc30-a6c4-4ded-8823-2e2a886b1d1c', vmId='c64d80f9-a3bd-44f3-89c4-f1b9b509cdde'}', device='ide', type='CONTROLLER', bootOrder='0', specParams='[]', address='{slot=0x01, bus=0x00, domain=0x0000, type=pci, function=0x1}', managed='false', plugged='true', readOnly='false', deviceAlias='ide', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_49fdfc30-a6c4-4ded-8823-2e2a886b1d1cdevice_cd110ea5-af7e-4e69-b906-c6f564873497device_873d0a5e-ca29-4710-adcd-c00bfd3145fedevice_b067852e-27a5-4598-934b-ad14fe031334=VmDevice:{id='VmDeviceId:{deviceId='b067852e-27a5-4598-934b-ad14fe031334', vmId='c64d80f9-a3bd-44f3-89c4-f1b9b509cdde'}', device='spicevmc', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=3}', managed='false', plugged='true', readOnly='false', deviceAlias='channel2', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_49fdfc30-a6c4-4ded-8823-2e2a886b1d1cdevice_cd110ea5-af7e-4e69-b906-c6f564873497device_873d0a5e-ca29-4710-adcd-c00bfd3145fe=VmDevice:{id='VmDeviceId:{deviceId='873d0a5e-ca29-4710-adcd-c00bfd3145fe', vmId='c64d80f9-a3bd-44f3-89c4-f1b9b509cdde'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=2}', managed='false', plugged='true', readOnly='false', deviceAlias='channel1', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_49fdfc30-a6c4-4ded-8823-2e2a886b1d1cdevice_cd110ea5-af7e-4e69-b906-c6f564873497=VmDevice:{id='VmDeviceId:{deviceId='cd110ea5-af7e-4e69-b906-c6f564873497', vmId='c64d80f9-a3bd-44f3-89c4-f1b9b509cdde'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=1}', managed='false', plugged='true', readOnly='false', deviceAlias='channel0', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}}, vmType=kvm, memSize=20480, smpCoresPerSocket=1, vmName=red, nice=0, guestFQDN=, maxMemSize=81920, bootMenuEnable=true, vmId=c64d80f9-a3bd-44f3-89c4-f1b9b509cdde, smpThreadsPerCore=1, guestDiskMapping={}, guestIPs=, kvmEnable=true, displayNetwork=yorku, devices=[Ljava.lang.Object;@788dc5c6, display=qxl, status=Paused, maxVCpus=16, clientIp=, statusTime=5260181460, maxMemSlots=16}], log id: 45ae4519 2017-10-01 15:52:45,088-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler3) [6c5c82d6] Received a memballoon Device without an address when processing VM c64d80f9-a3bd-44f3-89c4-f1b9b509cdde devices, skipping device: {device=memballoon, specParams={model=none}, type=balloon, target=20971520, alias=balloon0} 2017-10-01 15:52:45,089-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler3) [6c5c82d6] Received a spice Device without an address when processing VM c64d80f9-a3bd-44f3-89c4-f1b9b509cdde devices, skipping device: {device=spice, specParams={fileTransferEnable=true, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,ssmartcard,susbredir, copyPasteEnable=true, displayIp=130.63.94.36, displayNetwork=yorku}, type=graphics, deviceId=3462fd5e-740f-48bf-b77c-a44d530900a4, tlsPort=5903} 2017-10-01 15:52:45,580-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [7a77ffc9] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt1 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:52:45,580-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [7a77ffc9] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 15:52:45,580-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [7a77ffc9] HostName = virt1 2017-10-01 15:52:45,580-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [7a77ffc9] Command 'ConnectStoragePoolVDSCommand(HostName = virt1, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vdsId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:52:45,580-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [7a77ffc9] FINISH, ConnectStoragePoolVDSCommand, log id: 189e881f 2017-10-01 15:52:45,580-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler8) [7a77ffc9] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:52:45,692-04 INFO [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-25) [180aa754] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: c8653dba-63ec-4447-872c-0ffcc6af9c0f Type: Storage 2017-10-01 15:52:45,709-04 WARN [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-25) [180aa754] skipping connect and refresh for all hosts, last master 'true', command status 'true' 2017-10-01 15:52:45,715-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-25) [180aa754] EVENT_ID: RECONSTRUCT_MASTER_FAILED(985), Correlation ID: 180aa754, Job ID: 16c1c854-e2f1-4924-8e9b-f23666133ee5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center EECS-Primary. 2017-10-01 15:52:45,718-04 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-6-thread-20) [7a77ffc9] Finished reconstruct for pool 'aca61eac-ed8a-484b-bbee-3645148f6405'. Clearing event queue 2017-10-01 15:52:46,678-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [330ecfca] START, GetHardwareInfoVDSCommand(HostName = virt3, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vds='Host[virt3,e1e71e9d-a0cd-47db-96c6-f724f6cea736]'}), log id: 6af478df 2017-10-01 15:52:47,682-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler6) [330ecfca] FINISH, GetHardwareInfoVDSCommand, log id: 6af478df 2017-10-01 15:52:47,774-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler6) [5325cf30] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: e1e71e9d-a0cd-47db-96c6-f724f6cea736 Type: VDS 2017-10-01 15:52:47,796-04 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler6) [5325cf30] Cancelling the recovery from crash timer for VDS 'e1e71e9d-a0cd-47db-96c6-f724f6cea736' because vds started initializing 2017-10-01 15:52:47,800-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler6) [172370f6] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: aca61eac-ed8a-484b-bbee-3645148f6405 Type: StoragePool 2017-10-01 15:52:47,806-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [411dbd4b] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: aca61eac-ed8a-484b-bbee-3645148f6405 Type: StoragePool 2017-10-01 15:52:47,819-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [411dbd4b] START, ConnectStorageServerVDSCommand(HostName = virt3, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', storageType='NFS', connectionList='[StorageServerConnections:{id='5e557659-6817-40fa-9a55-7458125404a6', connection='virtfs-fs:/nfs/data', iqn='null', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='13cf1e67-9961-4a25-a41f-cc76b4ab793d', connection='virtfs-fs:/nfs/iso', iqn='null', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='417fdd51-dbcd-41d9-9a05-ef4d9f485d13', connection='virtfs-fs:/nfs/export', iqn='null', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 6018a5ac 2017-10-01 15:52:48,913-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [411dbd4b] FINISH, ConnectStorageServerVDSCommand, return: {5e557659-6817-40fa-9a55-7458125404a6=0, 417fdd51-dbcd-41d9-9a05-ef4d9f485d13=0, 13cf1e67-9961-4a25-a41f-cc76b4ab793d=0}, log id: 6018a5ac 2017-10-01 15:52:48,974-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler6) [411dbd4b] Host 'virt3' storage connection was succeeded 2017-10-01 15:52:48,984-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] START, ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 40999fc9 2017-10-01 15:52:48,984-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 15:52:50,100-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-15) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:52:50,100-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 15:52:50,100-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] HostName = virt3 2017-10-01 15:52:50,100-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] Command 'ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:52:50,101-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-15) [] FINISH, ConnectStoragePoolVDSCommand, log id: 40999fc9 2017-10-01 15:52:50,101-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-15) [] Could not connect host 'virt3' to pool 'EECS-Primary', as the master domain is in inactive/unknown status - not failing the operation 2017-10-01 15:52:51,305-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [411dbd4b] EVENT_ID: FENCE_OPERATION_USING_AGENT_AND_PROXY_STARTED(9,020), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Executing power management status on Host virt3 using Proxy Host virt1 and Fence Agent ipmilan:virt3-idrac. 2017-10-01 15:52:51,305-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler6) [411dbd4b] START, FenceVdsVDSCommand(HostName = virt1, FenceVdsVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', targetVdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', action='STATUS', agent='FenceAgent:{id='8c639beb-546a-4e77-9c21-1ffb54f1daae', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', order='1', type='ipmilan', ip='virt3-idrac', port='null', user='root', password='***', encryptOptions='false', options='privlvl=OPERATOR, delay=10, lanplus=1'}', policy='null'}), log id: 3a1faa37 2017-10-01 15:52:52,427-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler6) [411dbd4b] FINISH, FenceVdsVDSCommand, return: FenceOperationResult:{status='SUCCESS', powerStatus='ON', message=''}, log id: 3a1faa37 2017-10-01 15:52:52,442-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler6) [411dbd4b] START, SetMOMPolicyParametersVDSCommand(HostName = virt3, MomPolicyVDSParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736'}), log id: 529daca4 2017-10-01 15:52:53,489-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler6) [411dbd4b] FINISH, SetMOMPolicyParametersVDSCommand, log id: 529daca4 2017-10-01 15:52:53,507-04 INFO [org.ovirt.engine.core.bll.hostdev.RefreshHostDevicesCommand] (DefaultQuartzScheduler6) [533ad2d1] Running command: RefreshHostDevicesCommand internal: true. Entities affected : ID: e1e71e9d-a0cd-47db-96c6-f724f6cea736 Type: VDSAction group MANIPULATE_HOST with role type ADMIN 2017-10-01 15:52:53,661-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [533ad2d1] EVENT_ID: VDS_DETECTED(13), Correlation ID: 172370f6, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Status of host virt3 was set to Up. 2017-10-01 15:52:53,664-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [533ad2d1] EVENT_ID: VDS_FENCE_STATUS(496), Correlation ID: 172370f6, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host virt3 power management was verified successfully. 2017-10-01 15:52:53,714-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler6) [16521927] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: e1e71e9d-a0cd-47db-96c6-f724f6cea736 Type: VDS 2017-10-01 15:52:53,719-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-27) [] Domain 'c8653dba-63ec-4447-872c-0ffcc6af9c0f:data' recovered from problem. vds: 'virt3' 2017-10-01 15:52:53,719-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-27) [] Domain 'c8653dba-63ec-4447-872c-0ffcc6af9c0f:data' has recovered from problem. No active host in the DC is reporting it as problematic, so clearing the domain recovery timer. 2017-10-01 15:52:56,054-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [269332ae] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 15:52:56,694-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [1c4ec9fd] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt1 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 15:52:56,694-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler6) [1c4ec9fd] Command 'SpmStatusVDSCommand(HostName = virt1, SpmStatusVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 15:52:57,704-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [1c4ec9fd] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt1 command HSMGetAllTasksStatusesVDS failed: Not SPM 2017-10-01 15:52:57,704-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler6) [1c4ec9fd] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand' return value ' 2017-10-01 15:52:57,705-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler6) [1c4ec9fd] HostName = virt1 2017-10-01 15:52:57,705-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler6) [1c4ec9fd] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = virt1, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836'})' execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM 2017-10-01 15:52:57,767-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [1c4ec9fd] START, ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 532af940 2017-10-01 15:52:57,767-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [1c4ec9fd] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 15:52:58,218-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [1c4ec9fd] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:52:58,218-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [1c4ec9fd] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 15:52:58,218-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [1c4ec9fd] HostName = virt4 2017-10-01 15:52:58,218-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [1c4ec9fd] Command 'ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:52:58,218-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [1c4ec9fd] FINISH, ConnectStoragePoolVDSCommand, log id: 532af940 2017-10-01 15:52:58,218-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler6) [1c4ec9fd] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:52:59,337-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [1c4ec9fd] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 15:52:59,337-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler6) [1c4ec9fd] Command 'SpmStatusVDSCommand(HostName = virt4, SpmStatusVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 15:52:59,337-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler6) [1c4ec9fd] Irs placed on server '5ae29ed2-9c0f-4754-8df3-b43d14726846' failed. Proceed Failover 2017-10-01 15:52:59,401-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [1c4ec9fd] START, ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 3c8db6eb 2017-10-01 15:52:59,401-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [1c4ec9fd] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 15:52:59,664-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [1c4ec9fd] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:52:59,664-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [1c4ec9fd] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 15:52:59,664-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [1c4ec9fd] HostName = virt3 2017-10-01 15:52:59,664-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [1c4ec9fd] Command 'ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:52:59,664-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [1c4ec9fd] FINISH, ConnectStoragePoolVDSCommand, log id: 3c8db6eb 2017-10-01 15:52:59,664-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler6) [1c4ec9fd] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:52:59,873-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler6) [1c4ec9fd] Irs placed on server 'e1e71e9d-a0cd-47db-96c6-f724f6cea736' failed. Proceed Failover 2017-10-01 15:52:59,931-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [1c4ec9fd] START, ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: e88bd6f 2017-10-01 15:52:59,931-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [1c4ec9fd] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 15:53:00,106-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [1c4ec9fd] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:53:00,106-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [1c4ec9fd] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 15:53:00,106-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [1c4ec9fd] HostName = virt2 2017-10-01 15:53:00,106-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [1c4ec9fd] Command 'ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:53:00,106-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [1c4ec9fd] FINISH, ConnectStoragePoolVDSCommand, log id: e88bd6f 2017-10-01 15:53:00,106-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler6) [1c4ec9fd] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:53:01,226-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [1c4ec9fd] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 15:53:01,226-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler6) [1c4ec9fd] Command 'SpmStatusVDSCommand(HostName = virt2, SpmStatusVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 15:53:01,226-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler6) [1c4ec9fd] Irs placed on server 'a0ccfd11-3402-4eba-a119-4987561b738e' failed. Proceed Failover 2017-10-01 15:53:01,284-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [1c4ec9fd] START, ConnectStoragePoolVDSCommand(HostName = virt1, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vdsId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 6abeaab4 2017-10-01 15:53:01,284-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [1c4ec9fd] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 15:53:01,642-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [1c4ec9fd] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt1 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:53:01,642-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [1c4ec9fd] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 15:53:01,642-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [1c4ec9fd] HostName = virt1 2017-10-01 15:53:01,642-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [1c4ec9fd] Command 'ConnectStoragePoolVDSCommand(HostName = virt1, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vdsId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:53:01,642-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [1c4ec9fd] FINISH, ConnectStoragePoolVDSCommand, log id: 6abeaab4 2017-10-01 15:53:01,642-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler6) [1c4ec9fd] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:53:01,769-04 INFO [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-45) [53c1ed3a] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: c8653dba-63ec-4447-872c-0ffcc6af9c0f Type: Storage 2017-10-01 15:53:01,779-04 WARN [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-45) [53c1ed3a] skipping connect and refresh for all hosts, last master 'true', command status 'true' 2017-10-01 15:53:01,785-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-45) [53c1ed3a] EVENT_ID: RECONSTRUCT_MASTER_FAILED(985), Correlation ID: 53c1ed3a, Job ID: 3fec80f1-0aa7-4a25-add4-0c81ee2e75ec, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to Reconstruct Master Domain for Data Center EECS-Primary. 2017-10-01 15:53:01,788-04 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-6-thread-47) [1c4ec9fd] Finished reconstruct for pool 'aca61eac-ed8a-484b-bbee-3645148f6405'. Clearing event queue 2017-10-01 15:53:12,765-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [2d7bf685] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt1 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 15:53:12,765-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler2) [2d7bf685] Command 'SpmStatusVDSCommand(HostName = virt1, SpmStatusVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 15:53:13,418-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [2d7bf685] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt1 command HSMGetAllTasksStatusesVDS failed: Not SPM 2017-10-01 15:53:13,418-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler2) [2d7bf685] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand' return value ' 2017-10-01 15:53:13,418-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler2) [2d7bf685] HostName = virt1 2017-10-01 15:53:13,418-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler2) [2d7bf685] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = virt1, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836'})' execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM 2017-10-01 15:53:13,480-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [2d7bf685] START, ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 43ef5915 2017-10-01 15:53:13,480-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [2d7bf685] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 15:53:14,594-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [2d7bf685] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:53:14,594-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [2d7bf685] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 15:53:14,594-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [2d7bf685] HostName = virt3 2017-10-01 15:53:14,594-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [2d7bf685] Command 'ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:53:14,594-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [2d7bf685] FINISH, ConnectStoragePoolVDSCommand, log id: 43ef5915 2017-10-01 15:53:14,594-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler2) [2d7bf685] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:53:14,978-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler2) [2d7bf685] Irs placed on server 'e1e71e9d-a0cd-47db-96c6-f724f6cea736' failed. Proceed Failover 2017-10-01 15:53:15,048-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [2d7bf685] START, ConnectStoragePoolVDSCommand(HostName = virt1, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vdsId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 4f754bf2 2017-10-01 15:53:15,048-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [2d7bf685] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 15:53:15,229-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [2d7bf685] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt1 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:53:15,229-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [2d7bf685] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 15:53:15,229-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [2d7bf685] HostName = virt1 2017-10-01 15:53:15,229-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [2d7bf685] Command 'ConnectStoragePoolVDSCommand(HostName = virt1, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vdsId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:53:15,229-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [2d7bf685] FINISH, ConnectStoragePoolVDSCommand, log id: 4f754bf2 2017-10-01 15:53:15,229-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler2) [2d7bf685] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:53:16,361-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [2d7bf685] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt1 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 15:53:16,361-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler2) [2d7bf685] Command 'SpmStatusVDSCommand(HostName = virt1, SpmStatusVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 15:53:16,361-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler2) [2d7bf685] Irs placed on server '45f8b331-842e-48e7-9df8-56adddb93836' failed. Proceed Failover 2017-10-01 15:53:16,425-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [2d7bf685] START, ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 66095dfe 2017-10-01 15:53:16,425-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [2d7bf685] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 15:53:16,670-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [2d7bf685] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:53:16,670-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [2d7bf685] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 15:53:16,670-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [2d7bf685] HostName = virt2 2017-10-01 15:53:16,670-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [2d7bf685] Command 'ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:53:16,670-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [2d7bf685] FINISH, ConnectStoragePoolVDSCommand, log id: 66095dfe 2017-10-01 15:53:16,670-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler2) [2d7bf685] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:53:17,793-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [2d7bf685] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 15:53:17,793-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler2) [2d7bf685] Command 'SpmStatusVDSCommand(HostName = virt2, SpmStatusVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 15:53:17,793-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler2) [2d7bf685] Irs placed on server 'a0ccfd11-3402-4eba-a119-4987561b738e' failed. Proceed Failover 2017-10-01 15:53:17,850-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [2d7bf685] START, ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 791bfdf2 2017-10-01 15:53:17,850-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [2d7bf685] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 15:53:18,937-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [2d7bf685] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:53:18,937-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [2d7bf685] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 15:53:18,937-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [2d7bf685] HostName = virt4 2017-10-01 15:53:18,937-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [2d7bf685] Command 'ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:53:18,937-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [2d7bf685] FINISH, ConnectStoragePoolVDSCommand, log id: 791bfdf2 2017-10-01 15:53:18,937-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler2) [2d7bf685] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:53:19,065-04 INFO [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-18) [48925993] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: c8653dba-63ec-4447-872c-0ffcc6af9c0f Type: Storage 2017-10-01 15:53:19,077-04 WARN [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-18) [48925993] skipping connect and refresh for all hosts, last master 'true', command status 'true' 2017-10-01 15:53:19,086-04 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-6-thread-50) [2d7bf685] Finished reconstruct for pool 'aca61eac-ed8a-484b-bbee-3645148f6405'. Clearing event queue 2017-10-01 15:53:29,714-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2030d16f] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 15:53:29,715-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler8) [2030d16f] Command 'SpmStatusVDSCommand(HostName = virt4, SpmStatusVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 15:53:30,077-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2030d16f] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command HSMGetAllTasksStatusesVDS failed: Not SPM 2017-10-01 15:53:30,077-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler8) [2030d16f] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand' return value ' 2017-10-01 15:53:30,077-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler8) [2030d16f] HostName = virt4 2017-10-01 15:53:30,077-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler8) [2030d16f] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = virt4, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846'})' execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM 2017-10-01 15:53:30,139-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [2030d16f] START, ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 5d4a9320 2017-10-01 15:53:30,139-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [2030d16f] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 15:53:30,359-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2030d16f] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:53:30,359-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [2030d16f] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 15:53:30,359-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [2030d16f] HostName = virt2 2017-10-01 15:53:30,359-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [2030d16f] Command 'ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:53:30,359-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [2030d16f] FINISH, ConnectStoragePoolVDSCommand, log id: 5d4a9320 2017-10-01 15:53:30,359-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler8) [2030d16f] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:53:31,479-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2030d16f] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 15:53:31,479-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler8) [2030d16f] Command 'SpmStatusVDSCommand(HostName = virt2, SpmStatusVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 15:53:31,479-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler8) [2030d16f] Irs placed on server 'a0ccfd11-3402-4eba-a119-4987561b738e' failed. Proceed Failover 2017-10-01 15:53:31,536-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [2030d16f] START, ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 35ca36fe 2017-10-01 15:53:31,536-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [2030d16f] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 15:53:31,699-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2030d16f] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:53:31,699-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [2030d16f] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 15:53:31,699-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [2030d16f] HostName = virt3 2017-10-01 15:53:31,699-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [2030d16f] Command 'ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:53:31,699-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [2030d16f] FINISH, ConnectStoragePoolVDSCommand, log id: 35ca36fe 2017-10-01 15:53:31,699-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler8) [2030d16f] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:53:32,812-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler8) [2030d16f] Irs placed on server 'e1e71e9d-a0cd-47db-96c6-f724f6cea736' failed. Proceed Failover 2017-10-01 15:53:32,882-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [2030d16f] START, ConnectStoragePoolVDSCommand(HostName = virt1, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vdsId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 6bf5b5e2 2017-10-01 15:53:32,882-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [2030d16f] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 15:53:33,987-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2030d16f] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt1 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:53:33,987-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [2030d16f] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 15:53:33,987-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [2030d16f] HostName = virt1 2017-10-01 15:53:33,987-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [2030d16f] Command 'ConnectStoragePoolVDSCommand(HostName = virt1, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vdsId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:53:33,988-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [2030d16f] FINISH, ConnectStoragePoolVDSCommand, log id: 6bf5b5e2 2017-10-01 15:53:33,988-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler8) [2030d16f] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:53:35,116-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2030d16f] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt1 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 15:53:35,116-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler8) [2030d16f] Command 'SpmStatusVDSCommand(HostName = virt1, SpmStatusVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 15:53:35,116-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler8) [2030d16f] Irs placed on server '45f8b331-842e-48e7-9df8-56adddb93836' failed. Proceed Failover 2017-10-01 15:53:35,179-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [2030d16f] START, ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 36f0139 2017-10-01 15:53:35,179-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [2030d16f] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 15:53:36,263-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2030d16f] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:53:36,263-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [2030d16f] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 15:53:36,263-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [2030d16f] HostName = virt4 2017-10-01 15:53:36,263-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [2030d16f] Command 'ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:53:36,263-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [2030d16f] FINISH, ConnectStoragePoolVDSCommand, log id: 36f0139 2017-10-01 15:53:36,263-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler8) [2030d16f] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:53:36,400-04 INFO [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-32) [211019c8] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: c8653dba-63ec-4447-872c-0ffcc6af9c0f Type: Storage 2017-10-01 15:53:36,467-04 WARN [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-32) [211019c8] skipping connect and refresh for all hosts, last master 'true', command status 'true' 2017-10-01 15:53:36,481-04 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-6-thread-49) [2030d16f] Finished reconstruct for pool 'aca61eac-ed8a-484b-bbee-3645148f6405'. Clearing event queue 2017-10-01 15:53:46,718-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [3b774458] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 15:53:46,718-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler9) [3b774458] Command 'SpmStatusVDSCommand(HostName = virt4, SpmStatusVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 15:53:46,745-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [3b774458] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command HSMGetAllTasksStatusesVDS failed: Not SPM 2017-10-01 15:53:46,745-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler9) [3b774458] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand' return value ' 2017-10-01 15:53:46,745-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler9) [3b774458] HostName = virt4 2017-10-01 15:53:46,745-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler9) [3b774458] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = virt4, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846'})' execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM 2017-10-01 15:53:46,864-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [3b774458] START, ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 2f069b48 2017-10-01 15:53:46,864-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [3b774458] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 15:53:47,972-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [3b774458] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:53:47,972-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [3b774458] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 15:53:47,972-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [3b774458] HostName = virt3 2017-10-01 15:53:47,973-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [3b774458] Command 'ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:53:47,973-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [3b774458] FINISH, ConnectStoragePoolVDSCommand, log id: 2f069b48 2017-10-01 15:53:47,973-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler9) [3b774458] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:53:49,085-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler9) [3b774458] Irs placed on server 'e1e71e9d-a0cd-47db-96c6-f724f6cea736' failed. Proceed Failover 2017-10-01 15:53:49,143-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [3b774458] START, ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 7e852c3 2017-10-01 15:53:49,143-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [3b774458] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 15:53:50,235-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [3b774458] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:53:50,235-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [3b774458] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 15:53:50,235-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [3b774458] HostName = virt2 2017-10-01 15:53:50,235-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [3b774458] Command 'ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:53:50,235-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [3b774458] FINISH, ConnectStoragePoolVDSCommand, log id: 7e852c3 2017-10-01 15:53:50,235-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler9) [3b774458] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:53:51,354-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [3b774458] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 15:53:51,354-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler9) [3b774458] Command 'SpmStatusVDSCommand(HostName = virt2, SpmStatusVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 15:53:51,354-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler9) [3b774458] Irs placed on server 'a0ccfd11-3402-4eba-a119-4987561b738e' failed. Proceed Failover 2017-10-01 15:53:51,411-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [3b774458] START, ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 4e9eb071 2017-10-01 15:53:51,412-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [3b774458] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 15:53:52,497-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [3b774458] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:53:52,497-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [3b774458] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 15:53:52,497-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [3b774458] HostName = virt4 2017-10-01 15:53:52,497-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [3b774458] Command 'ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:53:52,497-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [3b774458] FINISH, ConnectStoragePoolVDSCommand, log id: 4e9eb071 2017-10-01 15:53:52,497-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler9) [3b774458] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:53:53,616-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [3b774458] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 15:53:53,616-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler9) [3b774458] Command 'SpmStatusVDSCommand(HostName = virt4, SpmStatusVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 15:53:53,616-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler9) [3b774458] Irs placed on server '5ae29ed2-9c0f-4754-8df3-b43d14726846' failed. Proceed Failover 2017-10-01 15:53:53,673-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [3b774458] START, ConnectStoragePoolVDSCommand(HostName = virt1, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vdsId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 2d5abd0e 2017-10-01 15:53:53,673-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [3b774458] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 15:53:54,773-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [3b774458] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt1 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:53:54,773-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [3b774458] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 15:53:54,773-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [3b774458] HostName = virt1 2017-10-01 15:53:54,773-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [3b774458] Command 'ConnectStoragePoolVDSCommand(HostName = virt1, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vdsId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:53:54,773-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [3b774458] FINISH, ConnectStoragePoolVDSCommand, log id: 2d5abd0e 2017-10-01 15:53:54,773-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler9) [3b774458] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:53:54,904-04 INFO [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-46) [4ee2226b] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: c8653dba-63ec-4447-872c-0ffcc6af9c0f Type: Storage 2017-10-01 15:53:54,912-04 WARN [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-46) [4ee2226b] skipping connect and refresh for all hosts, last master 'true', command status 'true' 2017-10-01 15:53:54,922-04 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-6-thread-11) [3b774458] Finished reconstruct for pool 'aca61eac-ed8a-484b-bbee-3645148f6405'. Clearing event queue 2017-10-01 15:53:56,074-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler7) [62ad3c31] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 15:54:05,897-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [61f386dc] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt1 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 15:54:05,897-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler1) [61f386dc] Command 'SpmStatusVDSCommand(HostName = virt1, SpmStatusVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 15:54:06,907-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [61f386dc] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt1 command HSMGetAllTasksStatusesVDS failed: Not SPM 2017-10-01 15:54:06,907-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler1) [61f386dc] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand' return value ' 2017-10-01 15:54:06,907-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler1) [61f386dc] HostName = virt1 2017-10-01 15:54:06,907-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler1) [61f386dc] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = virt1, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836'})' execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM 2017-10-01 15:54:06,969-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [61f386dc] START, ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 7c45ae5b 2017-10-01 15:54:06,969-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [61f386dc] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 15:54:08,052-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [61f386dc] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:54:08,052-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [61f386dc] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 15:54:08,052-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [61f386dc] HostName = virt4 2017-10-01 15:54:08,052-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [61f386dc] Command 'ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:54:08,052-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [61f386dc] FINISH, ConnectStoragePoolVDSCommand, log id: 7c45ae5b 2017-10-01 15:54:08,052-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler1) [61f386dc] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:54:09,171-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [61f386dc] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 15:54:09,171-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler1) [61f386dc] Command 'SpmStatusVDSCommand(HostName = virt4, SpmStatusVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 15:54:09,171-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler1) [61f386dc] Irs placed on server '5ae29ed2-9c0f-4754-8df3-b43d14726846' failed. Proceed Failover 2017-10-01 15:54:09,228-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [61f386dc] START, ConnectStoragePoolVDSCommand(HostName = virt1, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vdsId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 4319db95 2017-10-01 15:54:09,228-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [61f386dc] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 15:54:10,326-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [61f386dc] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt1 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:54:10,326-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [61f386dc] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 15:54:10,326-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [61f386dc] HostName = virt1 2017-10-01 15:54:10,326-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [61f386dc] Command 'ConnectStoragePoolVDSCommand(HostName = virt1, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vdsId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:54:10,326-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [61f386dc] FINISH, ConnectStoragePoolVDSCommand, log id: 4319db95 2017-10-01 15:54:10,326-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler1) [61f386dc] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:54:11,446-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [61f386dc] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt1 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 15:54:11,446-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler1) [61f386dc] Command 'SpmStatusVDSCommand(HostName = virt1, SpmStatusVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 15:54:11,446-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler1) [61f386dc] Irs placed on server '45f8b331-842e-48e7-9df8-56adddb93836' failed. Proceed Failover 2017-10-01 15:54:11,503-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [61f386dc] START, ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 669cac 2017-10-01 15:54:11,503-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [61f386dc] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 15:54:12,192-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [61f386dc] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:54:12,192-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [61f386dc] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 15:54:12,192-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [61f386dc] HostName = virt3 2017-10-01 15:54:12,192-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [61f386dc] Command 'ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:54:12,192-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [61f386dc] FINISH, ConnectStoragePoolVDSCommand, log id: 669cac 2017-10-01 15:54:12,192-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler1) [61f386dc] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:54:13,305-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler1) [61f386dc] Irs placed on server 'e1e71e9d-a0cd-47db-96c6-f724f6cea736' failed. Proceed Failover 2017-10-01 15:54:13,363-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [61f386dc] START, ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 4f12c671 2017-10-01 15:54:13,363-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [61f386dc] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 15:54:14,460-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [61f386dc] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:54:14,460-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [61f386dc] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 15:54:14,460-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [61f386dc] HostName = virt2 2017-10-01 15:54:14,460-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [61f386dc] Command 'ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:54:14,460-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [61f386dc] FINISH, ConnectStoragePoolVDSCommand, log id: 4f12c671 2017-10-01 15:54:14,460-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler1) [61f386dc] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:54:14,586-04 INFO [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-27) [59b2aa9f] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: c8653dba-63ec-4447-872c-0ffcc6af9c0f Type: Storage 2017-10-01 15:54:14,615-04 WARN [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-27) [59b2aa9f] skipping connect and refresh for all hosts, last master 'true', command status 'true' 2017-10-01 15:54:14,628-04 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-6-thread-10) [61f386dc] Finished reconstruct for pool 'aca61eac-ed8a-484b-bbee-3645148f6405'. Clearing event queue 2017-10-01 15:54:25,581-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [7a700b7f] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 15:54:25,581-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler9) [7a700b7f] Command 'SpmStatusVDSCommand(HostName = virt2, SpmStatusVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 15:54:26,589-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [7a700b7f] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command HSMGetAllTasksStatusesVDS failed: Not SPM 2017-10-01 15:54:26,589-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler9) [7a700b7f] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand' return value ' 2017-10-01 15:54:26,589-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler9) [7a700b7f] HostName = virt2 2017-10-01 15:54:26,589-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler9) [7a700b7f] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = virt2, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e'})' execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM 2017-10-01 15:54:26,650-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [7a700b7f] START, ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 103dddee 2017-10-01 15:54:26,651-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [7a700b7f] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 15:54:27,393-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [7a700b7f] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:54:27,393-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [7a700b7f] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 15:54:27,393-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [7a700b7f] HostName = virt2 2017-10-01 15:54:27,393-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [7a700b7f] Command 'ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:54:27,393-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [7a700b7f] FINISH, ConnectStoragePoolVDSCommand, log id: 103dddee 2017-10-01 15:54:27,393-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler9) [7a700b7f] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:54:28,514-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [7a700b7f] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 15:54:28,514-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler9) [7a700b7f] Command 'SpmStatusVDSCommand(HostName = virt2, SpmStatusVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 15:54:28,514-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler9) [7a700b7f] Irs placed on server 'a0ccfd11-3402-4eba-a119-4987561b738e' failed. Proceed Failover 2017-10-01 15:54:28,572-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [7a700b7f] START, ConnectStoragePoolVDSCommand(HostName = virt1, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vdsId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 65b633d2 2017-10-01 15:54:28,572-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [7a700b7f] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 15:54:29,668-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [7a700b7f] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt1 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:54:29,668-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [7a700b7f] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 15:54:29,668-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [7a700b7f] HostName = virt1 2017-10-01 15:54:29,668-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [7a700b7f] Command 'ConnectStoragePoolVDSCommand(HostName = virt1, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vdsId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:54:29,668-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [7a700b7f] FINISH, ConnectStoragePoolVDSCommand, log id: 65b633d2 2017-10-01 15:54:29,668-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler9) [7a700b7f] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:54:29,824-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [7a700b7f] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt1 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 15:54:29,824-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler9) [7a700b7f] Command 'SpmStatusVDSCommand(HostName = virt1, SpmStatusVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 15:54:29,824-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler9) [7a700b7f] Irs placed on server '45f8b331-842e-48e7-9df8-56adddb93836' failed. Proceed Failover 2017-10-01 15:54:29,881-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [7a700b7f] START, ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 3186a324 2017-10-01 15:54:29,882-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [7a700b7f] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 15:54:30,555-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [7a700b7f] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:54:30,556-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [7a700b7f] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 15:54:30,556-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [7a700b7f] HostName = virt4 2017-10-01 15:54:30,556-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [7a700b7f] Command 'ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:54:30,556-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [7a700b7f] FINISH, ConnectStoragePoolVDSCommand, log id: 3186a324 2017-10-01 15:54:30,556-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler9) [7a700b7f] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:54:31,585-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [7a700b7f] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 15:54:31,585-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler9) [7a700b7f] Command 'SpmStatusVDSCommand(HostName = virt4, SpmStatusVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 15:54:31,585-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler9) [7a700b7f] Irs placed on server '5ae29ed2-9c0f-4754-8df3-b43d14726846' failed. Proceed Failover 2017-10-01 15:54:31,642-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [7a700b7f] START, ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 68537cfb 2017-10-01 15:54:31,642-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [7a700b7f] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 15:54:31,779-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [7a700b7f] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:54:31,779-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [7a700b7f] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 15:54:31,779-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [7a700b7f] HostName = virt3 2017-10-01 15:54:31,779-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [7a700b7f] Command 'ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:54:31,779-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [7a700b7f] FINISH, ConnectStoragePoolVDSCommand, log id: 68537cfb 2017-10-01 15:54:31,779-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler9) [7a700b7f] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:54:31,927-04 INFO [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-43) [515b3d4e] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: c8653dba-63ec-4447-872c-0ffcc6af9c0f Type: Storage 2017-10-01 15:54:32,003-04 WARN [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-43) [515b3d4e] skipping connect and refresh for all hosts, last master 'true', command status 'true' 2017-10-01 15:54:32,016-04 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-6-thread-37) [7a700b7f] Finished reconstruct for pool 'aca61eac-ed8a-484b-bbee-3645148f6405'. Clearing event queue 2017-10-01 15:54:44,515-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [7d6a0c2c] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command HSMGetAllTasksStatusesVDS failed: Not SPM: () 2017-10-01 15:54:44,515-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler5) [7d6a0c2c] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand' return value ' 2017-10-01 15:54:44,515-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler5) [7d6a0c2c] HostName = virt3 2017-10-01 15:54:44,515-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler5) [7d6a0c2c] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = virt3, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736'})' execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () 2017-10-01 15:54:44,583-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [7d6a0c2c] START, ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 3c84d158 2017-10-01 15:54:44,583-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [7d6a0c2c] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 15:54:44,855-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [7d6a0c2c] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:54:44,855-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [7d6a0c2c] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 15:54:44,855-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [7d6a0c2c] HostName = virt3 2017-10-01 15:54:44,855-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [7d6a0c2c] Command 'ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:54:44,855-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [7d6a0c2c] FINISH, ConnectStoragePoolVDSCommand, log id: 3c84d158 2017-10-01 15:54:44,855-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler5) [7d6a0c2c] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:54:45,748-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler5) [7d6a0c2c] Irs placed on server 'e1e71e9d-a0cd-47db-96c6-f724f6cea736' failed. Proceed Failover 2017-10-01 15:54:45,806-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [7d6a0c2c] START, ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 301a3894 2017-10-01 15:54:45,806-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [7d6a0c2c] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 15:54:46,736-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [7d6a0c2c] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:54:46,736-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [7d6a0c2c] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 15:54:46,736-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [7d6a0c2c] HostName = virt2 2017-10-01 15:54:46,736-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [7d6a0c2c] Command 'ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:54:46,736-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [7d6a0c2c] FINISH, ConnectStoragePoolVDSCommand, log id: 301a3894 2017-10-01 15:54:46,736-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler5) [7d6a0c2c] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:54:46,959-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [7d6a0c2c] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 15:54:46,959-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler5) [7d6a0c2c] Command 'SpmStatusVDSCommand(HostName = virt2, SpmStatusVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 15:54:46,959-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler5) [7d6a0c2c] Irs placed on server 'a0ccfd11-3402-4eba-a119-4987561b738e' failed. Proceed Failover 2017-10-01 15:54:47,015-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [7d6a0c2c] START, ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 7f13f7f5 2017-10-01 15:54:47,016-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [7d6a0c2c] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 15:54:48,099-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [7d6a0c2c] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:54:48,099-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [7d6a0c2c] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 15:54:48,099-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [7d6a0c2c] HostName = virt4 2017-10-01 15:54:48,099-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [7d6a0c2c] Command 'ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:54:48,099-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [7d6a0c2c] FINISH, ConnectStoragePoolVDSCommand, log id: 7f13f7f5 2017-10-01 15:54:48,099-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler5) [7d6a0c2c] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:54:49,218-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [7d6a0c2c] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 15:54:49,218-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler5) [7d6a0c2c] Command 'SpmStatusVDSCommand(HostName = virt4, SpmStatusVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 15:54:49,218-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler5) [7d6a0c2c] Irs placed on server '5ae29ed2-9c0f-4754-8df3-b43d14726846' failed. Proceed Failover 2017-10-01 15:54:49,281-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [7d6a0c2c] START, ConnectStoragePoolVDSCommand(HostName = virt1, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vdsId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 4874d689 2017-10-01 15:54:49,282-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [7d6a0c2c] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 15:54:50,378-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [7d6a0c2c] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt1 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:54:50,378-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [7d6a0c2c] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 15:54:50,378-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [7d6a0c2c] HostName = virt1 2017-10-01 15:54:50,378-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [7d6a0c2c] Command 'ConnectStoragePoolVDSCommand(HostName = virt1, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vdsId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:54:50,378-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [7d6a0c2c] FINISH, ConnectStoragePoolVDSCommand, log id: 4874d689 2017-10-01 15:54:50,378-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler5) [7d6a0c2c] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:54:50,508-04 INFO [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-35) [37454a29] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: c8653dba-63ec-4447-872c-0ffcc6af9c0f Type: Storage 2017-10-01 15:54:50,573-04 WARN [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-35) [37454a29] skipping connect and refresh for all hosts, last master 'true', command status 'true' 2017-10-01 15:54:50,588-04 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-6-thread-36) [7d6a0c2c] Finished reconstruct for pool 'aca61eac-ed8a-484b-bbee-3645148f6405'. Clearing event queue 2017-10-01 15:54:56,094-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [321d0570] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 15:55:00,014-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [7198ec5f] Autorecovering 1 storage domains 2017-10-01 15:55:00,014-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [7198ec5f] Autorecovering storage domains id: c8653dba-63ec-4447-872c-0ffcc6af9c0f 2017-10-01 15:55:00,018-04 INFO [org.ovirt.engine.core.bll.storage.connection.ConnectDomainToStorageCommand] (DefaultQuartzScheduler6) [4c858f13] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: c8653dba-63ec-4447-872c-0ffcc6af9c0f Type: Storage 2017-10-01 15:55:00,018-04 INFO [org.ovirt.engine.core.bll.storage.connection.ConnectDomainToStorageCommand] (DefaultQuartzScheduler6) [4c858f13] ConnectDomainToStorage. Before Connect all hosts to pool. Time: Sun Oct 01 15:55:00 EDT 2017 2017-10-01 15:55:00,081-04 INFO [org.ovirt.engine.core.bll.storage.connection.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-6-thread-49) [7048f5a6] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN 2017-10-01 15:55:00,083-04 INFO [org.ovirt.engine.core.bll.storage.connection.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-6-thread-32) [5b9ace44] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN 2017-10-01 15:55:00,083-04 INFO [org.ovirt.engine.core.bll.storage.connection.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-6-thread-2) [2ffab463] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN 2017-10-01 15:55:00,084-04 INFO [org.ovirt.engine.core.bll.storage.connection.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-6-thread-40) [5fa41d2a] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN 2017-10-01 15:55:00,099-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-32) [5b9ace44] START, ConnectStorageServerVDSCommand(HostName = virt1, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='00000000-0000-0000-0000-000000000000', storageType='NFS', connectionList='[StorageServerConnections:{id='5e557659-6817-40fa-9a55-7458125404a6', connection='virtfs-fs:/nfs/data', iqn='null', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 63c112c5 2017-10-01 15:55:00,099-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-2) [2ffab463] START, ConnectStorageServerVDSCommand(HostName = virt2, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='00000000-0000-0000-0000-000000000000', storageType='NFS', connectionList='[StorageServerConnections:{id='5e557659-6817-40fa-9a55-7458125404a6', connection='virtfs-fs:/nfs/data', iqn='null', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 34b60c8a 2017-10-01 15:55:00,099-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-40) [5fa41d2a] START, ConnectStorageServerVDSCommand(HostName = virt4, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='00000000-0000-0000-0000-000000000000', storageType='NFS', connectionList='[StorageServerConnections:{id='5e557659-6817-40fa-9a55-7458125404a6', connection='virtfs-fs:/nfs/data', iqn='null', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 6651cf12 2017-10-01 15:55:00,113-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-40) [5fa41d2a] FINISH, ConnectStorageServerVDSCommand, return: {5e557659-6817-40fa-9a55-7458125404a6=0}, log id: 6651cf12 2017-10-01 15:55:00,115-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-2) [2ffab463] FINISH, ConnectStorageServerVDSCommand, return: {5e557659-6817-40fa-9a55-7458125404a6=0}, log id: 34b60c8a 2017-10-01 15:55:00,117-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-49) [7048f5a6] START, ConnectStorageServerVDSCommand(HostName = virt3, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='00000000-0000-0000-0000-000000000000', storageType='NFS', connectionList='[StorageServerConnections:{id='5e557659-6817-40fa-9a55-7458125404a6', connection='virtfs-fs:/nfs/data', iqn='null', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 4fca9ef7 2017-10-01 15:55:00,119-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-32) [5b9ace44] FINISH, ConnectStorageServerVDSCommand, return: {5e557659-6817-40fa-9a55-7458125404a6=0}, log id: 63c112c5 2017-10-01 15:55:00,496-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-49) [7048f5a6] FINISH, ConnectStorageServerVDSCommand, return: {5e557659-6817-40fa-9a55-7458125404a6=0}, log id: 4fca9ef7 2017-10-01 15:55:00,499-04 INFO [org.ovirt.engine.core.bll.storage.connection.ConnectDomainToStorageCommand] (DefaultQuartzScheduler6) [4c858f13] ConnectDomainToStorage. After Connect all hosts to pool. Time: Sun Oct 01 15:55:00 EDT 2017 2017-10-01 15:55:00,909-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [d6bc803] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt1 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 15:55:00,909-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler10) [d6bc803] Command 'SpmStatusVDSCommand(HostName = virt1, SpmStatusVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 15:55:01,822-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [d6bc803] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt1 command HSMGetAllTasksStatusesVDS failed: Not SPM 2017-10-01 15:55:01,822-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler10) [d6bc803] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand' return value ' 2017-10-01 15:55:01,822-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler10) [d6bc803] HostName = virt1 2017-10-01 15:55:01,822-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler10) [d6bc803] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = virt1, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836'})' execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM 2017-10-01 15:55:01,955-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [d6bc803] START, ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 513f24e5 2017-10-01 15:55:01,955-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [d6bc803] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 15:55:02,109-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [d6bc803] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:55:02,109-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [d6bc803] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 15:55:02,109-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [d6bc803] HostName = virt4 2017-10-01 15:55:02,109-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [d6bc803] Command 'ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:55:02,109-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [d6bc803] FINISH, ConnectStoragePoolVDSCommand, log id: 513f24e5 2017-10-01 15:55:02,109-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler10) [d6bc803] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:55:03,228-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [d6bc803] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 15:55:03,228-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler10) [d6bc803] Command 'SpmStatusVDSCommand(HostName = virt4, SpmStatusVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 15:55:03,228-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler10) [d6bc803] Irs placed on server '5ae29ed2-9c0f-4754-8df3-b43d14726846' failed. Proceed Failover 2017-10-01 15:55:03,286-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [d6bc803] START, ConnectStoragePoolVDSCommand(HostName = virt1, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vdsId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 40b365fe 2017-10-01 15:55:03,286-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [d6bc803] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 15:55:04,391-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [d6bc803] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt1 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:55:04,391-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [d6bc803] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 15:55:04,391-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [d6bc803] HostName = virt1 2017-10-01 15:55:04,391-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [d6bc803] Command 'ConnectStoragePoolVDSCommand(HostName = virt1, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vdsId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:55:04,391-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [d6bc803] FINISH, ConnectStoragePoolVDSCommand, log id: 40b365fe 2017-10-01 15:55:04,391-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler10) [d6bc803] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:55:05,526-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [d6bc803] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt1 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 15:55:05,526-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler10) [d6bc803] Command 'SpmStatusVDSCommand(HostName = virt1, SpmStatusVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 15:55:05,526-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler10) [d6bc803] Irs placed on server '45f8b331-842e-48e7-9df8-56adddb93836' failed. Proceed Failover 2017-10-01 15:55:05,596-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [d6bc803] START, ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 14fc7155 2017-10-01 15:55:05,596-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [d6bc803] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 15:55:06,716-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [d6bc803] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:55:06,716-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [d6bc803] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 15:55:06,716-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [d6bc803] HostName = virt3 2017-10-01 15:55:06,716-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [d6bc803] Command 'ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:55:06,716-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [d6bc803] FINISH, ConnectStoragePoolVDSCommand, log id: 14fc7155 2017-10-01 15:55:06,716-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler10) [d6bc803] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:55:07,830-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler10) [d6bc803] Irs placed on server 'e1e71e9d-a0cd-47db-96c6-f724f6cea736' failed. Proceed Failover 2017-10-01 15:55:07,887-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [d6bc803] START, ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 1dd519d1 2017-10-01 15:55:07,888-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [d6bc803] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 15:55:08,982-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [d6bc803] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:55:08,982-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [d6bc803] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 15:55:08,982-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [d6bc803] HostName = virt2 2017-10-01 15:55:08,982-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [d6bc803] Command 'ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:55:08,982-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [d6bc803] FINISH, ConnectStoragePoolVDSCommand, log id: 1dd519d1 2017-10-01 15:55:08,982-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler10) [d6bc803] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:55:09,116-04 INFO [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-11) [3622ed2c] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: c8653dba-63ec-4447-872c-0ffcc6af9c0f Type: Storage 2017-10-01 15:55:09,185-04 WARN [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-11) [3622ed2c] skipping connect and refresh for all hosts, last master 'true', command status 'true' 2017-10-01 15:55:09,196-04 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-6-thread-46) [d6bc803] Finished reconstruct for pool 'aca61eac-ed8a-484b-bbee-3645148f6405'. Clearing event queue 2017-10-01 15:55:20,112-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [42a0cd2] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 15:55:20,112-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler9) [42a0cd2] Command 'SpmStatusVDSCommand(HostName = virt2, SpmStatusVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 15:55:21,120-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [42a0cd2] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command HSMGetAllTasksStatusesVDS failed: Not SPM 2017-10-01 15:55:21,120-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler9) [42a0cd2] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand' return value ' 2017-10-01 15:55:21,120-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler9) [42a0cd2] HostName = virt2 2017-10-01 15:55:21,120-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler9) [42a0cd2] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = virt2, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e'})' execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM 2017-10-01 15:55:21,193-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [42a0cd2] START, ConnectStoragePoolVDSCommand(HostName = virt1, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vdsId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 16468374 2017-10-01 15:55:21,193-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [42a0cd2] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 15:55:22,302-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [42a0cd2] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt1 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:55:22,302-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [42a0cd2] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 15:55:22,302-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [42a0cd2] HostName = virt1 2017-10-01 15:55:22,302-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [42a0cd2] Command 'ConnectStoragePoolVDSCommand(HostName = virt1, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vdsId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:55:22,302-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [42a0cd2] FINISH, ConnectStoragePoolVDSCommand, log id: 16468374 2017-10-01 15:55:22,302-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler9) [42a0cd2] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:55:23,434-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [42a0cd2] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt1 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 15:55:23,434-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler9) [42a0cd2] Command 'SpmStatusVDSCommand(HostName = virt1, SpmStatusVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 15:55:23,434-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler9) [42a0cd2] Irs placed on server '45f8b331-842e-48e7-9df8-56adddb93836' failed. Proceed Failover 2017-10-01 15:55:23,497-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [42a0cd2] START, ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 4b0bd3c0 2017-10-01 15:55:23,497-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [42a0cd2] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 15:55:24,609-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [42a0cd2] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:55:24,609-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [42a0cd2] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 15:55:24,609-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [42a0cd2] HostName = virt3 2017-10-01 15:55:24,609-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [42a0cd2] Command 'ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:55:24,609-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [42a0cd2] FINISH, ConnectStoragePoolVDSCommand, log id: 4b0bd3c0 2017-10-01 15:55:24,609-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler9) [42a0cd2] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:55:25,722-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler9) [42a0cd2] Irs placed on server 'e1e71e9d-a0cd-47db-96c6-f724f6cea736' failed. Proceed Failover 2017-10-01 15:55:25,780-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [42a0cd2] START, ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 2d5929c7 2017-10-01 15:55:25,780-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [42a0cd2] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 15:55:29,143-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [42a0cd2] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:55:29,143-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [42a0cd2] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 15:55:29,143-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [42a0cd2] HostName = virt2 2017-10-01 15:55:29,143-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [42a0cd2] Command 'ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:55:29,143-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [42a0cd2] FINISH, ConnectStoragePoolVDSCommand, log id: 2d5929c7 2017-10-01 15:55:29,143-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler9) [42a0cd2] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:55:29,800-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [42a0cd2] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 15:55:29,800-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler9) [42a0cd2] Command 'SpmStatusVDSCommand(HostName = virt2, SpmStatusVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 15:55:29,800-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler9) [42a0cd2] Irs placed on server 'a0ccfd11-3402-4eba-a119-4987561b738e' failed. Proceed Failover 2017-10-01 15:55:29,858-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [42a0cd2] START, ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 130e9571 2017-10-01 15:55:29,858-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [42a0cd2] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 15:55:30,956-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [42a0cd2] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:55:30,956-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [42a0cd2] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 15:55:30,956-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [42a0cd2] HostName = virt4 2017-10-01 15:55:30,956-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [42a0cd2] Command 'ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:55:30,956-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [42a0cd2] FINISH, ConnectStoragePoolVDSCommand, log id: 130e9571 2017-10-01 15:55:30,956-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler9) [42a0cd2] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:55:31,082-04 INFO [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-10) [448249f7] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: c8653dba-63ec-4447-872c-0ffcc6af9c0f Type: Storage 2017-10-01 15:55:31,097-04 WARN [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-10) [448249f7] skipping connect and refresh for all hosts, last master 'true', command status 'true' 2017-10-01 15:55:31,113-04 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-6-thread-27) [42a0cd2] Finished reconstruct for pool 'aca61eac-ed8a-484b-bbee-3645148f6405'. Clearing event queue 2017-10-01 15:55:42,078-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [1fe62244] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 15:55:42,078-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler10) [1fe62244] Command 'SpmStatusVDSCommand(HostName = virt4, SpmStatusVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 15:55:43,086-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [1fe62244] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command HSMGetAllTasksStatusesVDS failed: Not SPM 2017-10-01 15:55:43,086-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler10) [1fe62244] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand' return value ' 2017-10-01 15:55:43,086-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler10) [1fe62244] HostName = virt4 2017-10-01 15:55:43,086-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler10) [1fe62244] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = virt4, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846'})' execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM 2017-10-01 15:55:43,212-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [1fe62244] START, ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 7e960292 2017-10-01 15:55:43,212-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [1fe62244] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 15:55:44,293-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [1fe62244] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:55:44,293-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [1fe62244] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 15:55:44,293-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [1fe62244] HostName = virt4 2017-10-01 15:55:44,293-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [1fe62244] Command 'ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:55:44,293-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [1fe62244] FINISH, ConnectStoragePoolVDSCommand, log id: 7e960292 2017-10-01 15:55:44,294-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler10) [1fe62244] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:55:44,952-04 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to virt3-mgmt/192.168.0.36 2017-10-01 15:55:44,970-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [1fe62244] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 15:55:44,970-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler10) [1fe62244] Command 'SpmStatusVDSCommand(HostName = virt4, SpmStatusVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 15:55:44,970-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler10) [1fe62244] Irs placed on server '5ae29ed2-9c0f-4754-8df3-b43d14726846' failed. Proceed Failover 2017-10-01 15:55:45,027-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [1fe62244] START, ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 7d567f8b 2017-10-01 15:55:45,027-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [1fe62244] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 15:55:46,919-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [1fe62244] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:55:46,919-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [1fe62244] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 15:55:46,919-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [1fe62244] HostName = virt3 2017-10-01 15:55:46,919-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [1fe62244] Command 'ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:55:46,919-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [1fe62244] FINISH, ConnectStoragePoolVDSCommand, log id: 7d567f8b 2017-10-01 15:55:46,919-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler10) [1fe62244] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:55:47,169-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler10) [1fe62244] Irs placed on server 'e1e71e9d-a0cd-47db-96c6-f724f6cea736' failed. Proceed Failover 2017-10-01 15:55:47,226-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [1fe62244] START, ConnectStoragePoolVDSCommand(HostName = virt1, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vdsId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 3f6987a8 2017-10-01 15:55:47,226-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [1fe62244] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 15:55:48,323-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [1fe62244] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt1 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:55:48,323-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [1fe62244] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 15:55:48,323-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [1fe62244] HostName = virt1 2017-10-01 15:55:48,323-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [1fe62244] Command 'ConnectStoragePoolVDSCommand(HostName = virt1, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vdsId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:55:48,323-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [1fe62244] FINISH, ConnectStoragePoolVDSCommand, log id: 3f6987a8 2017-10-01 15:55:48,323-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler10) [1fe62244] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:55:49,445-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [1fe62244] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt1 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 15:55:49,445-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler10) [1fe62244] Command 'SpmStatusVDSCommand(HostName = virt1, SpmStatusVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 15:55:49,445-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler10) [1fe62244] Irs placed on server '45f8b331-842e-48e7-9df8-56adddb93836' failed. Proceed Failover 2017-10-01 15:55:49,508-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [1fe62244] START, ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 10dc7e8d 2017-10-01 15:55:49,508-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [1fe62244] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 15:55:50,602-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [1fe62244] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:55:50,602-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [1fe62244] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 15:55:50,602-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [1fe62244] HostName = virt2 2017-10-01 15:55:50,602-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [1fe62244] Command 'ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:55:50,602-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [1fe62244] FINISH, ConnectStoragePoolVDSCommand, log id: 10dc7e8d 2017-10-01 15:55:50,602-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler10) [1fe62244] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:55:50,743-04 INFO [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-14) [598cc9cf] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: c8653dba-63ec-4447-872c-0ffcc6af9c0f Type: Storage 2017-10-01 15:55:50,756-04 WARN [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-14) [598cc9cf] skipping connect and refresh for all hosts, last master 'true', command status 'true' 2017-10-01 15:55:50,764-04 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-6-thread-4) [1fe62244] Finished reconstruct for pool 'aca61eac-ed8a-484b-bbee-3645148f6405'. Clearing event queue 2017-10-01 15:55:56,113-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [30c99ea3] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 15:56:01,222-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [64171eec] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 15:56:01,222-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler1) [64171eec] Command 'SpmStatusVDSCommand(HostName = virt2, SpmStatusVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 15:56:01,944-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [64171eec] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command HSMGetAllTasksStatusesVDS failed: Not SPM 2017-10-01 15:56:01,944-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler1) [64171eec] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand' return value ' 2017-10-01 15:56:01,944-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler1) [64171eec] HostName = virt2 2017-10-01 15:56:01,944-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler1) [64171eec] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = virt2, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e'})' execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM 2017-10-01 15:56:02,062-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [64171eec] START, ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 6b486b1b 2017-10-01 15:56:02,062-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [64171eec] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 15:56:02,269-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [64171eec] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:56:02,270-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [64171eec] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 15:56:02,270-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [64171eec] HostName = virt4 2017-10-01 15:56:02,270-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [64171eec] Command 'ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:56:02,270-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [64171eec] FINISH, ConnectStoragePoolVDSCommand, log id: 6b486b1b 2017-10-01 15:56:02,270-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler1) [64171eec] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:56:03,393-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [64171eec] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 15:56:03,393-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler1) [64171eec] Command 'SpmStatusVDSCommand(HostName = virt4, SpmStatusVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 15:56:03,393-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler1) [64171eec] Irs placed on server '5ae29ed2-9c0f-4754-8df3-b43d14726846' failed. Proceed Failover 2017-10-01 15:56:03,451-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [64171eec] START, ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 8f1685b 2017-10-01 15:56:03,451-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [64171eec] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 15:56:04,566-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [64171eec] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:56:04,566-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [64171eec] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 15:56:04,566-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [64171eec] HostName = virt3 2017-10-01 15:56:04,566-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [64171eec] Command 'ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:56:04,566-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [64171eec] FINISH, ConnectStoragePoolVDSCommand, log id: 8f1685b 2017-10-01 15:56:04,566-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler1) [64171eec] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:56:04,682-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler1) [64171eec] Irs placed on server 'e1e71e9d-a0cd-47db-96c6-f724f6cea736' failed. Proceed Failover 2017-10-01 15:56:04,740-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [64171eec] START, ConnectStoragePoolVDSCommand(HostName = virt1, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vdsId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 7fbc2353 2017-10-01 15:56:04,740-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [64171eec] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 15:56:05,835-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [64171eec] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt1 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:56:05,835-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [64171eec] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 15:56:05,835-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [64171eec] HostName = virt1 2017-10-01 15:56:05,835-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [64171eec] Command 'ConnectStoragePoolVDSCommand(HostName = virt1, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vdsId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:56:05,835-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [64171eec] FINISH, ConnectStoragePoolVDSCommand, log id: 7fbc2353 2017-10-01 15:56:05,835-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler1) [64171eec] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:56:06,955-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [64171eec] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt1 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 15:56:06,955-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler1) [64171eec] Command 'SpmStatusVDSCommand(HostName = virt1, SpmStatusVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 15:56:06,955-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler1) [64171eec] Irs placed on server '45f8b331-842e-48e7-9df8-56adddb93836' failed. Proceed Failover 2017-10-01 15:56:07,012-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [64171eec] START, ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 55455d7d 2017-10-01 15:56:07,013-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [64171eec] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 15:56:08,105-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [64171eec] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:56:08,105-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [64171eec] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 15:56:08,105-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [64171eec] HostName = virt2 2017-10-01 15:56:08,105-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [64171eec] Command 'ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:56:08,105-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [64171eec] FINISH, ConnectStoragePoolVDSCommand, log id: 55455d7d 2017-10-01 15:56:08,105-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler1) [64171eec] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:56:08,231-04 INFO [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-41) [7b423038] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: c8653dba-63ec-4447-872c-0ffcc6af9c0f Type: Storage 2017-10-01 15:56:08,242-04 WARN [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-41) [7b423038] skipping connect and refresh for all hosts, last master 'true', command status 'true' 2017-10-01 15:56:08,249-04 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-6-thread-24) [64171eec] Finished reconstruct for pool 'aca61eac-ed8a-484b-bbee-3645148f6405'. Clearing event queue 2017-10-01 15:56:19,226-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [cd20a20] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 15:56:19,226-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler6) [cd20a20] Command 'SpmStatusVDSCommand(HostName = virt2, SpmStatusVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 15:56:19,831-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [cd20a20] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command HSMGetAllTasksStatusesVDS failed: Not SPM 2017-10-01 15:56:19,831-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler6) [cd20a20] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand' return value ' 2017-10-01 15:56:19,831-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler6) [cd20a20] HostName = virt2 2017-10-01 15:56:19,831-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler6) [cd20a20] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = virt2, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e'})' execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM 2017-10-01 15:56:19,893-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [cd20a20] START, ConnectStoragePoolVDSCommand(HostName = virt1, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vdsId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 61c0fef9 2017-10-01 15:56:19,893-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [cd20a20] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 15:56:20,990-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [cd20a20] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt1 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:56:20,990-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [cd20a20] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 15:56:20,990-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [cd20a20] HostName = virt1 2017-10-01 15:56:20,990-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [cd20a20] Command 'ConnectStoragePoolVDSCommand(HostName = virt1, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vdsId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:56:20,990-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [cd20a20] FINISH, ConnectStoragePoolVDSCommand, log id: 61c0fef9 2017-10-01 15:56:20,990-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler6) [cd20a20] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:56:22,110-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [cd20a20] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt1 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 15:56:22,110-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler6) [cd20a20] Command 'SpmStatusVDSCommand(HostName = virt1, SpmStatusVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 15:56:22,110-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler6) [cd20a20] Irs placed on server '45f8b331-842e-48e7-9df8-56adddb93836' failed. Proceed Failover 2017-10-01 15:56:22,174-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [cd20a20] START, ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 74b3c6a8 2017-10-01 15:56:22,174-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [cd20a20] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 15:56:23,266-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [cd20a20] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:56:23,266-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [cd20a20] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 15:56:23,266-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [cd20a20] HostName = virt2 2017-10-01 15:56:23,266-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [cd20a20] Command 'ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:56:23,266-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [cd20a20] FINISH, ConnectStoragePoolVDSCommand, log id: 74b3c6a8 2017-10-01 15:56:23,266-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler6) [cd20a20] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:56:23,324-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-44) [] starting processDomainRecovery for domain '63d680c7-4c94-4b29-a6cb-f288a128b9bc:iso'. 2017-10-01 15:56:23,379-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-44) [] Domain '63d680c7-4c94-4b29-a6cb-f288a128b9bc:iso' was reported by all hosts in status UP as problematic. Moving the domain to NonOperational. 2017-10-01 15:56:23,393-04 INFO [org.ovirt.engine.core.bll.storage.domain.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-6-thread-44) [347afa51] Lock Acquired to object 'EngineLock:{exclusiveLocks='[63d680c7-4c94-4b29-a6cb-f288a128b9bc=STORAGE]', sharedLocks=''}' 2017-10-01 15:56:23,398-04 INFO [org.ovirt.engine.core.bll.storage.domain.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-6-thread-44) [347afa51] Running command: DeactivateStorageDomainCommand internal: true. Entities affected : ID: 63d680c7-4c94-4b29-a6cb-f288a128b9bc Type: StorageAction group MANIPULATE_STORAGE_DOMAIN with role type ADMIN 2017-10-01 15:56:23,400-04 INFO [org.ovirt.engine.core.bll.storage.domain.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-6-thread-44) [347afa51] Lock freed to object 'EngineLock:{exclusiveLocks='[63d680c7-4c94-4b29-a6cb-f288a128b9bc=STORAGE]', sharedLocks=''}' 2017-10-01 15:56:23,406-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-44) [347afa51] EVENT_ID: SYSTEM_DEACTIVATED_STORAGE_DOMAIN(970), Correlation ID: 347afa51, Job ID: adce5dc6-90f7-498e-84ca-b4a452dd526c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Storage Domain iso (Data Center EECS-Primary) was deactivated by system because it's not visible by any of the hosts. 2017-10-01 15:56:23,409-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-44) [347afa51] Removing vds '[]' from the domain in maintenance cache 2017-10-01 15:56:23,409-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-44) [347afa51] Removing host(s) '[]' from hosts unseen domain report cache 2017-10-01 15:56:23,409-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-11) [] starting processDomainRecovery for domain 'f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2:export'. 2017-10-01 15:56:23,464-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-11) [] Domain 'f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2:export' was reported by all hosts in status UP as problematic. Moving the domain to NonOperational. 2017-10-01 15:56:23,475-04 INFO [org.ovirt.engine.core.bll.storage.domain.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-6-thread-11) [4d0d09d1] Lock Acquired to object 'EngineLock:{exclusiveLocks='[f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=STORAGE]', sharedLocks=''}' 2017-10-01 15:56:23,480-04 INFO [org.ovirt.engine.core.bll.storage.domain.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-6-thread-11) [4d0d09d1] Running command: DeactivateStorageDomainCommand internal: true. Entities affected : ID: f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2 Type: StorageAction group MANIPULATE_STORAGE_DOMAIN with role type ADMIN 2017-10-01 15:56:23,483-04 INFO [org.ovirt.engine.core.bll.storage.domain.DeactivateStorageDomainCommand] (org.ovirt.thread.pool-6-thread-11) [4d0d09d1] Lock freed to object 'EngineLock:{exclusiveLocks='[f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=STORAGE]', sharedLocks=''}' 2017-10-01 15:56:23,488-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-11) [4d0d09d1] EVENT_ID: SYSTEM_DEACTIVATED_STORAGE_DOMAIN(970), Correlation ID: 4d0d09d1, Job ID: 8631ac2e-aea6-4efe-ac68-efd093b47bd3, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Storage Domain export (Data Center EECS-Primary) was deactivated by system because it's not visible by any of the hosts. 2017-10-01 15:56:23,490-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-11) [4d0d09d1] Removing vds '[]' from the domain in maintenance cache 2017-10-01 15:56:23,490-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-11) [4d0d09d1] Removing host(s) '[]' from hosts unseen domain report cache 2017-10-01 15:56:24,386-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [cd20a20] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 15:56:24,386-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler6) [cd20a20] Command 'SpmStatusVDSCommand(HostName = virt2, SpmStatusVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 15:56:24,386-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler6) [cd20a20] Irs placed on server 'a0ccfd11-3402-4eba-a119-4987561b738e' failed. Proceed Failover 2017-10-01 15:56:24,444-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [cd20a20] START, ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 54df1420 2017-10-01 15:56:24,444-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [cd20a20] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 15:56:25,552-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [cd20a20] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:56:25,552-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [cd20a20] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 15:56:25,552-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [cd20a20] HostName = virt3 2017-10-01 15:56:25,552-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [cd20a20] Command 'ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:56:25,552-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [cd20a20] FINISH, ConnectStoragePoolVDSCommand, log id: 54df1420 2017-10-01 15:56:25,552-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler6) [cd20a20] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:56:26,665-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler6) [cd20a20] Irs placed on server 'e1e71e9d-a0cd-47db-96c6-f724f6cea736' failed. Proceed Failover 2017-10-01 15:56:26,723-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [cd20a20] START, ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 4948e852 2017-10-01 15:56:26,723-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [cd20a20] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 15:56:27,805-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [cd20a20] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:56:27,805-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [cd20a20] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 15:56:27,805-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [cd20a20] HostName = virt4 2017-10-01 15:56:27,805-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [cd20a20] Command 'ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:56:27,805-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [cd20a20] FINISH, ConnectStoragePoolVDSCommand, log id: 4948e852 2017-10-01 15:56:27,805-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler6) [cd20a20] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:56:27,930-04 INFO [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-34) [40bd7328] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: c8653dba-63ec-4447-872c-0ffcc6af9c0f Type: Storage 2017-10-01 15:56:27,940-04 WARN [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-34) [40bd7328] skipping connect and refresh for all hosts, last master 'true', command status 'true' 2017-10-01 15:56:27,947-04 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-6-thread-46) [cd20a20] Finished reconstruct for pool 'aca61eac-ed8a-484b-bbee-3645148f6405'. Clearing event queue 2017-10-01 15:56:38,927-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [151ed11b] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 15:56:38,927-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler2) [151ed11b] Command 'SpmStatusVDSCommand(HostName = virt4, SpmStatusVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 15:56:39,935-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [151ed11b] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command HSMGetAllTasksStatusesVDS failed: Not SPM 2017-10-01 15:56:39,935-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler2) [151ed11b] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand' return value ' 2017-10-01 15:56:39,935-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler2) [151ed11b] HostName = virt4 2017-10-01 15:56:39,935-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler2) [151ed11b] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = virt4, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846'})' execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM 2017-10-01 15:56:40,002-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [151ed11b] START, ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 240735c 2017-10-01 15:56:40,002-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [151ed11b] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 15:56:41,112-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [151ed11b] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:56:41,112-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [151ed11b] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 15:56:41,112-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [151ed11b] HostName = virt3 2017-10-01 15:56:41,112-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [151ed11b] Command 'ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:56:41,112-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [151ed11b] FINISH, ConnectStoragePoolVDSCommand, log id: 240735c 2017-10-01 15:56:41,112-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler2) [151ed11b] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:56:42,225-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler2) [151ed11b] Irs placed on server 'e1e71e9d-a0cd-47db-96c6-f724f6cea736' failed. Proceed Failover 2017-10-01 15:56:42,288-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [151ed11b] START, ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 25f45b2b 2017-10-01 15:56:42,288-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [151ed11b] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 15:56:43,372-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [151ed11b] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:56:43,372-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [151ed11b] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 15:56:43,372-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [151ed11b] HostName = virt4 2017-10-01 15:56:43,372-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [151ed11b] Command 'ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:56:43,372-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [151ed11b] FINISH, ConnectStoragePoolVDSCommand, log id: 25f45b2b 2017-10-01 15:56:43,372-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler2) [151ed11b] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:56:44,491-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [151ed11b] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 15:56:44,491-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler2) [151ed11b] Command 'SpmStatusVDSCommand(HostName = virt4, SpmStatusVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 15:56:44,491-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler2) [151ed11b] Irs placed on server '5ae29ed2-9c0f-4754-8df3-b43d14726846' failed. Proceed Failover 2017-10-01 15:56:44,548-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [151ed11b] START, ConnectStoragePoolVDSCommand(HostName = virt1, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vdsId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 53b39c94 2017-10-01 15:56:44,548-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [151ed11b] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 15:56:45,077-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [151ed11b] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt1 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:56:45,077-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [151ed11b] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 15:56:45,077-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [151ed11b] HostName = virt1 2017-10-01 15:56:45,077-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [151ed11b] Command 'ConnectStoragePoolVDSCommand(HostName = virt1, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vdsId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:56:45,077-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [151ed11b] FINISH, ConnectStoragePoolVDSCommand, log id: 53b39c94 2017-10-01 15:56:45,077-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler2) [151ed11b] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 15:56:56,132-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [5f431a65] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 15:57:46,200-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [151ed11b] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt1 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 15:57:46,200-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler2) [151ed11b] Command 'SpmStatusVDSCommand(HostName = virt1, SpmStatusVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 15:57:46,200-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler2) [151ed11b] Irs placed on server '45f8b331-842e-48e7-9df8-56adddb93836' failed. Proceed Failover 2017-10-01 15:57:46,270-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [151ed11b] START, ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 3ef1998 2017-10-01 15:57:46,270-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [151ed11b] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 15:57:56,157-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [42bcd58a] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 15:58:17,229-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler4) [4529f623] VM 'db3e08fb-c111-442d-aeec-1507da30f486'(print) moved from 'Paused' --> 'NotResponding' 2017-10-01 15:58:17,300-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [4529f623] EVENT_ID: VM_NOT_RESPONDING(126), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM print is not responding. 2017-10-01 15:58:17,300-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler4) [4529f623] VM 'ae56c31c-71f7-433a-b198-a1c2c846325e'(forum) moved from 'Paused' --> 'NotResponding' 2017-10-01 15:58:17,321-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [4529f623] EVENT_ID: VM_NOT_RESPONDING(126), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM forum is not responding. 2017-10-01 15:58:17,322-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler4) [4529f623] VM 'ddf37076-e795-4c87-a4fb-9efcaae76d72'(dc1) moved from 'Paused' --> 'NotResponding' 2017-10-01 15:58:17,342-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [4529f623] EVENT_ID: VM_NOT_RESPONDING(126), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM dc1 is not responding. 2017-10-01 15:58:17,343-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler4) [4529f623] VM 'bfe9a524-2238-4bab-a91f-5dc6ed270896'(pcmode) moved from 'Paused' --> 'NotResponding' 2017-10-01 15:58:17,364-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [4529f623] EVENT_ID: VM_NOT_RESPONDING(126), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM pcmode is not responding. 2017-10-01 15:58:17,364-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler4) [4529f623] VM '1325cebf-834e-4571-a3e1-82beb0021ffe'(dns1) moved from 'Paused' --> 'NotResponding' 2017-10-01 15:58:17,385-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [4529f623] EVENT_ID: VM_NOT_RESPONDING(126), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM dns1 is not responding. 2017-10-01 15:58:17,385-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler4) [4529f623] VM 'a82b4cc3-6ef6-4a8e-97ef-1d4ea1b7e07c'(indigo) moved from 'Paused' --> 'NotResponding' 2017-10-01 15:58:17,406-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [4529f623] EVENT_ID: VM_NOT_RESPONDING(126), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM indigo is not responding. 2017-10-01 15:58:17,406-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler4) [4529f623] VM 'c8682859-9e55-460a-9ef0-d03e4a85f50e'(dhcp1) moved from 'Paused' --> 'NotResponding' 2017-10-01 15:58:17,427-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [4529f623] EVENT_ID: VM_NOT_RESPONDING(126), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM dhcp1 is not responding. 2017-10-01 15:58:17,427-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler4) [4529f623] VM '3bb83ad5-dc00-445d-8708-2db18c53e0e6'(www-vhost) moved from 'Paused' --> 'NotResponding' 2017-10-01 15:58:17,448-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [4529f623] EVENT_ID: VM_NOT_RESPONDING(126), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM www-vhost is not responding. 2017-10-01 15:58:17,448-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler4) [4529f623] VM 'a8d4e149-55ec-4961-a414-d7d1e1b05a59'(fix) moved from 'Paused' --> 'NotResponding' 2017-10-01 15:58:17,469-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [4529f623] EVENT_ID: VM_NOT_RESPONDING(126), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM fix is not responding. 2017-10-01 15:58:17,469-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler4) [4529f623] VM 'aa487207-7ff4-465a-9d9b-2a103d50dc77'(ftp1) moved from 'Paused' --> 'NotResponding' 2017-10-01 15:58:17,490-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [4529f623] EVENT_ID: VM_NOT_RESPONDING(126), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM ftp1 is not responding. 2017-10-01 15:58:17,490-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler4) [4529f623] VM '4ba80442-b361-4026-ae8c-895fd28042b0'(dist) moved from 'Paused' --> 'NotResponding' 2017-10-01 15:58:17,511-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [4529f623] EVENT_ID: VM_NOT_RESPONDING(126), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM dist is not responding. 2017-10-01 15:58:30,308-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler5) [3688c3e4] VM '07331c78-6e09-41b4-9b29-edda32f46351'(samba) moved from 'Paused' --> 'NotResponding' 2017-10-01 15:58:30,378-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [3688c3e4] EVENT_ID: VM_NOT_RESPONDING(126), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM samba is not responding. 2017-10-01 15:58:30,379-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler5) [3688c3e4] VM 'dc66c051-24b7-4cb8-99eb-db398d396ef1'(samba1) moved from 'Paused' --> 'NotResponding' 2017-10-01 15:58:30,400-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [3688c3e4] EVENT_ID: VM_NOT_RESPONDING(126), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM samba1 is not responding. 2017-10-01 15:58:30,400-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler5) [3688c3e4] VM 'bcd5776e-978b-4723-933d-70a80322a796'(cape) moved from 'Paused' --> 'NotResponding' 2017-10-01 15:58:30,421-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [3688c3e4] EVENT_ID: VM_NOT_RESPONDING(126), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM cape is not responding. 2017-10-01 15:58:30,421-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler5) [3688c3e4] VM '3e5a5728-8940-4fe7-b4af-11fb8e12f116'(www) moved from 'Paused' --> 'NotResponding' 2017-10-01 15:58:30,442-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [3688c3e4] EVENT_ID: VM_NOT_RESPONDING(126), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM www is not responding. 2017-10-01 15:58:30,443-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler5) [3688c3e4] VM 'f87ec08d-0fdf-42ba-8aac-c66c404d7aae'(webapp) moved from 'Paused' --> 'NotResponding' 2017-10-01 15:58:30,463-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [3688c3e4] EVENT_ID: VM_NOT_RESPONDING(126), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM webapp is not responding. 2017-10-01 15:58:30,463-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler5) [3688c3e4] VM '911d6883-607c-464e-b46b-8e6141bdd2de'(hdb) moved from 'Paused' --> 'NotResponding' 2017-10-01 15:58:30,484-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [3688c3e4] EVENT_ID: VM_NOT_RESPONDING(126), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM hdb is not responding. 2017-10-01 15:58:30,484-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler5) [3688c3e4] VM '8391f9a7-7fe7-4025-8fba-6273ae587a52'(archive) moved from 'Paused' --> 'NotResponding' 2017-10-01 15:58:30,505-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [3688c3e4] EVENT_ID: VM_NOT_RESPONDING(126), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM archive is not responding. 2017-10-01 15:58:30,505-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler5) [3688c3e4] VM '1c7e966f-29e8-4788-b64c-86e52f415061'(udb) moved from 'Paused' --> 'NotResponding' 2017-10-01 15:58:30,526-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [3688c3e4] EVENT_ID: VM_NOT_RESPONDING(126), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM udb is not responding. 2017-10-01 15:58:30,526-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler5) [3688c3e4] VM '9ea2821e-557f-45cb-9872-1e6c5aeaf246'(rs) moved from 'Paused' --> 'NotResponding' 2017-10-01 15:58:30,547-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [3688c3e4] EVENT_ID: VM_NOT_RESPONDING(126), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM rs is not responding. 2017-10-01 15:58:30,547-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler5) [3688c3e4] VM 'd66235b4-7c8d-4da2-a572-82ecc9c53a59'(www-las) moved from 'Paused' --> 'NotResponding' 2017-10-01 15:58:30,568-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [3688c3e4] EVENT_ID: VM_NOT_RESPONDING(126), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM www-las is not responding. 2017-10-01 15:58:30,568-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler5) [3688c3e4] VM '5885c43a-b133-45dc-b8d5-1aa2a4a19579'(prs) moved from 'Up' --> 'NotResponding' 2017-10-01 15:58:30,589-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [3688c3e4] EVENT_ID: VM_NOT_RESPONDING(126), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM prs is not responding. 2017-10-01 15:58:30,589-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler5) [3688c3e4] VM 'ae65a104-ed3f-40e0-ba92-5708cf338918'(vpn2) moved from 'Paused' --> 'NotResponding' 2017-10-01 15:58:30,610-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [3688c3e4] EVENT_ID: VM_NOT_RESPONDING(126), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM vpn2 is not responding. 2017-10-01 15:58:30,610-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler5) [3688c3e4] VM 'd56363ad-9cb1-4a70-a33a-ce439b044d7d'(svn) moved from 'Paused' --> 'NotResponding' 2017-10-01 15:58:30,631-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [3688c3e4] EVENT_ID: VM_NOT_RESPONDING(126), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM svn is not responding. 2017-10-01 15:58:32,534-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler6) [7a496af8] VM 'db3e08fb-c111-442d-aeec-1507da30f486'(print) moved from 'NotResponding' --> 'Paused' 2017-10-01 15:58:32,605-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [7a496af8] EVENT_ID: VM_PAUSED(1,025), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM print has been paused. 2017-10-01 15:58:32,626-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [7a496af8] EVENT_ID: VM_PAUSED_ERROR(139), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM print has been paused due to unknown storage error. 2017-10-01 15:58:32,626-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler6) [7a496af8] VM 'ae56c31c-71f7-433a-b198-a1c2c846325e'(forum) moved from 'NotResponding' --> 'Paused' 2017-10-01 15:58:32,647-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [7a496af8] EVENT_ID: VM_PAUSED(1,025), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM forum has been paused. 2017-10-01 15:58:32,668-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [7a496af8] EVENT_ID: VM_PAUSED_ERROR(139), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM forum has been paused due to unknown storage error. 2017-10-01 15:58:32,668-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler6) [7a496af8] VM 'ddf37076-e795-4c87-a4fb-9efcaae76d72'(dc1) moved from 'NotResponding' --> 'Paused' 2017-10-01 15:58:32,689-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [7a496af8] EVENT_ID: VM_PAUSED(1,025), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM dc1 has been paused. 2017-10-01 15:58:32,710-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [7a496af8] EVENT_ID: VM_PAUSED_ERROR(139), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM dc1 has been paused due to unknown storage error. 2017-10-01 15:58:32,710-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler6) [7a496af8] VM 'bfe9a524-2238-4bab-a91f-5dc6ed270896'(pcmode) moved from 'NotResponding' --> 'Paused' 2017-10-01 15:58:32,731-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [7a496af8] EVENT_ID: VM_PAUSED(1,025), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM pcmode has been paused. 2017-10-01 15:58:32,752-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [7a496af8] EVENT_ID: VM_PAUSED_ERROR(139), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM pcmode has been paused due to unknown storage error. 2017-10-01 15:58:32,752-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler6) [7a496af8] VM '1325cebf-834e-4571-a3e1-82beb0021ffe'(dns1) moved from 'NotResponding' --> 'Paused' 2017-10-01 15:58:32,773-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [7a496af8] EVENT_ID: VM_PAUSED(1,025), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM dns1 has been paused. 2017-10-01 15:58:32,793-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [7a496af8] EVENT_ID: VM_PAUSED_ERROR(139), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM dns1 has been paused due to unknown storage error. 2017-10-01 15:58:32,793-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler6) [7a496af8] VM 'a82b4cc3-6ef6-4a8e-97ef-1d4ea1b7e07c'(indigo) moved from 'NotResponding' --> 'Paused' 2017-10-01 15:58:32,814-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [7a496af8] EVENT_ID: VM_PAUSED(1,025), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM indigo has been paused. 2017-10-01 15:58:32,834-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [7a496af8] EVENT_ID: VM_PAUSED_ERROR(139), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM indigo has been paused due to unknown storage error. 2017-10-01 15:58:32,835-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler6) [7a496af8] VM 'c8682859-9e55-460a-9ef0-d03e4a85f50e'(dhcp1) moved from 'NotResponding' --> 'Paused' 2017-10-01 15:58:32,855-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [7a496af8] EVENT_ID: VM_PAUSED(1,025), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM dhcp1 has been paused. 2017-10-01 15:58:32,876-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [7a496af8] EVENT_ID: VM_PAUSED_ERROR(139), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM dhcp1 has been paused due to unknown storage error. 2017-10-01 15:58:32,876-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler6) [7a496af8] VM '3bb83ad5-dc00-445d-8708-2db18c53e0e6'(www-vhost) moved from 'NotResponding' --> 'Paused' 2017-10-01 15:58:32,896-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [7a496af8] EVENT_ID: VM_PAUSED(1,025), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM www-vhost has been paused. 2017-10-01 15:58:32,917-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [7a496af8] EVENT_ID: VM_PAUSED_ERROR(139), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM www-vhost has been paused due to unknown storage error. 2017-10-01 15:58:32,917-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler6) [7a496af8] VM 'a8d4e149-55ec-4961-a414-d7d1e1b05a59'(fix) moved from 'NotResponding' --> 'Paused' 2017-10-01 15:58:32,938-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [7a496af8] EVENT_ID: VM_PAUSED(1,025), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM fix has been paused. 2017-10-01 15:58:32,959-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [7a496af8] EVENT_ID: VM_PAUSED_ERROR(139), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM fix has been paused due to unknown storage error. 2017-10-01 15:58:32,959-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler6) [7a496af8] VM 'aa487207-7ff4-465a-9d9b-2a103d50dc77'(ftp1) moved from 'NotResponding' --> 'Paused' 2017-10-01 15:58:32,980-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [7a496af8] EVENT_ID: VM_PAUSED(1,025), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM ftp1 has been paused. 2017-10-01 15:58:33,000-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [7a496af8] EVENT_ID: VM_PAUSED_ERROR(139), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM ftp1 has been paused due to unknown storage error. 2017-10-01 15:58:33,000-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler6) [7a496af8] VM '4ba80442-b361-4026-ae8c-895fd28042b0'(dist) moved from 'NotResponding' --> 'Paused' 2017-10-01 15:58:33,021-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [7a496af8] EVENT_ID: VM_PAUSED(1,025), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM dist has been paused. 2017-10-01 15:58:33,041-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [7a496af8] EVENT_ID: VM_PAUSED_ERROR(139), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM dist has been paused due to unknown storage error. 2017-10-01 15:58:56,173-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [a2ae3c6] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 15:59:15,714-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler1) [21c736da] VM '07331c78-6e09-41b4-9b29-edda32f46351'(samba) moved from 'NotResponding' --> 'Paused' 2017-10-01 15:59:15,783-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [21c736da] EVENT_ID: VM_PAUSED(1,025), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM samba has been paused. 2017-10-01 15:59:15,805-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [21c736da] EVENT_ID: VM_PAUSED_ERROR(139), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM samba has been paused due to unknown storage error. 2017-10-01 15:59:15,805-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler1) [21c736da] VM 'dc66c051-24b7-4cb8-99eb-db398d396ef1'(samba1) moved from 'NotResponding' --> 'Paused' 2017-10-01 15:59:15,826-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [21c736da] EVENT_ID: VM_PAUSED(1,025), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM samba1 has been paused. 2017-10-01 15:59:15,847-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [21c736da] EVENT_ID: VM_PAUSED_ERROR(139), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM samba1 has been paused due to unknown storage error. 2017-10-01 15:59:15,847-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler1) [21c736da] VM 'bcd5776e-978b-4723-933d-70a80322a796'(cape) moved from 'NotResponding' --> 'Paused' 2017-10-01 15:59:15,868-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [21c736da] EVENT_ID: VM_PAUSED(1,025), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM cape has been paused. 2017-10-01 15:59:15,888-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [21c736da] EVENT_ID: VM_PAUSED_ERROR(139), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM cape has been paused due to unknown storage error. 2017-10-01 15:59:15,888-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler1) [21c736da] VM '3e5a5728-8940-4fe7-b4af-11fb8e12f116'(www) moved from 'NotResponding' --> 'Paused' 2017-10-01 15:59:15,908-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [21c736da] EVENT_ID: VM_PAUSED(1,025), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM www has been paused. 2017-10-01 15:59:15,929-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [21c736da] EVENT_ID: VM_PAUSED_ERROR(139), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM www has been paused due to unknown storage error. 2017-10-01 15:59:15,929-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler1) [21c736da] VM 'f87ec08d-0fdf-42ba-8aac-c66c404d7aae'(webapp) moved from 'NotResponding' --> 'Paused' 2017-10-01 15:59:15,950-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [21c736da] EVENT_ID: VM_PAUSED(1,025), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM webapp has been paused. 2017-10-01 15:59:15,970-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [21c736da] EVENT_ID: VM_PAUSED_ERROR(139), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM webapp has been paused due to unknown storage error. 2017-10-01 15:59:15,970-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler1) [21c736da] VM '911d6883-607c-464e-b46b-8e6141bdd2de'(hdb) moved from 'NotResponding' --> 'Paused' 2017-10-01 15:59:15,990-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [21c736da] EVENT_ID: VM_PAUSED(1,025), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM hdb has been paused. 2017-10-01 15:59:16,011-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [21c736da] EVENT_ID: VM_PAUSED_ERROR(139), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM hdb has been paused due to unknown storage error. 2017-10-01 15:59:16,011-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler1) [21c736da] VM '8391f9a7-7fe7-4025-8fba-6273ae587a52'(archive) moved from 'NotResponding' --> 'Paused' 2017-10-01 15:59:16,031-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [21c736da] EVENT_ID: VM_PAUSED(1,025), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM archive has been paused. 2017-10-01 15:59:16,051-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [21c736da] EVENT_ID: VM_PAUSED_ERROR(139), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM archive has been paused due to unknown storage error. 2017-10-01 15:59:16,051-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler1) [21c736da] VM '1c7e966f-29e8-4788-b64c-86e52f415061'(udb) moved from 'NotResponding' --> 'Paused' 2017-10-01 15:59:16,072-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [21c736da] EVENT_ID: VM_PAUSED(1,025), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM udb has been paused. 2017-10-01 15:59:16,093-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [21c736da] EVENT_ID: VM_PAUSED_ERROR(139), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM udb has been paused due to unknown storage error. 2017-10-01 15:59:16,093-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler1) [21c736da] VM '9ea2821e-557f-45cb-9872-1e6c5aeaf246'(rs) moved from 'NotResponding' --> 'Paused' 2017-10-01 15:59:16,114-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [21c736da] EVENT_ID: VM_PAUSED(1,025), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM rs has been paused. 2017-10-01 15:59:16,134-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [21c736da] EVENT_ID: VM_PAUSED_ERROR(139), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM rs has been paused due to unknown storage error. 2017-10-01 15:59:16,135-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler1) [21c736da] VM 'd66235b4-7c8d-4da2-a572-82ecc9c53a59'(www-las) moved from 'NotResponding' --> 'Paused' 2017-10-01 15:59:16,155-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [21c736da] EVENT_ID: VM_PAUSED(1,025), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM www-las has been paused. 2017-10-01 15:59:16,175-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [21c736da] EVENT_ID: VM_PAUSED_ERROR(139), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM www-las has been paused due to unknown storage error. 2017-10-01 15:59:16,175-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler1) [21c736da] VM '5885c43a-b133-45dc-b8d5-1aa2a4a19579'(prs) moved from 'NotResponding' --> 'Up' 2017-10-01 15:59:16,176-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler1) [21c736da] VM 'ae65a104-ed3f-40e0-ba92-5708cf338918'(vpn2) moved from 'NotResponding' --> 'Paused' 2017-10-01 15:59:16,196-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [21c736da] EVENT_ID: VM_PAUSED(1,025), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM vpn2 has been paused. 2017-10-01 15:59:16,217-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [21c736da] EVENT_ID: VM_PAUSED_ERROR(139), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM vpn2 has been paused due to unknown storage error. 2017-10-01 15:59:16,217-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler1) [21c736da] VM 'd56363ad-9cb1-4a70-a33a-ce439b044d7d'(svn) moved from 'NotResponding' --> 'Paused' 2017-10-01 15:59:16,237-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [21c736da] EVENT_ID: VM_PAUSED(1,025), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM svn has been paused. 2017-10-01 15:59:16,258-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [21c736da] EVENT_ID: VM_PAUSED_ERROR(139), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM svn has been paused due to unknown storage error. 2017-10-01 15:59:23,989-04 INFO [org.ovirt.engine.core.bll.StopVmCommand] (org.ovirt.thread.pool-6-thread-13) [f6c99088-149b-457e-a985-b2d5885c8a28] Running command: StopVmCommand internal: false. Entities affected : ID: bcd5776e-978b-4723-933d-70a80322a796 Type: VMAction group STOP_VM with role type USER 2017-10-01 15:59:24,008-04 INFO [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (org.ovirt.thread.pool-6-thread-13) [f6c99088-149b-457e-a985-b2d5885c8a28] START, DestroyVmVDSCommand( DestroyVmVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vmId='bcd5776e-978b-4723-933d-70a80322a796', force='false', secondsToWait='0', gracefully='false', reason='', ignoreNoVm='false'}), log id: 5179072d 2017-10-01 15:59:24,008-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-6-thread-13) [f6c99088-149b-457e-a985-b2d5885c8a28] START, DestroyVDSCommand(HostName = virt4, DestroyVmVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vmId='bcd5776e-978b-4723-933d-70a80322a796', force='false', secondsToWait='0', gracefully='false', reason='', ignoreNoVm='false'}), log id: 4788a20c 2017-10-01 15:59:33,143-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler7) [1b9834d1] VM 'db3e08fb-c111-442d-aeec-1507da30f486'(print) moved from 'Paused' --> 'NotResponding' 2017-10-01 15:59:33,213-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [1b9834d1] EVENT_ID: VM_NOT_RESPONDING(126), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM print is not responding. 2017-10-01 15:59:33,214-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler7) [1b9834d1] VM 'ae56c31c-71f7-433a-b198-a1c2c846325e'(forum) moved from 'Paused' --> 'NotResponding' 2017-10-01 15:59:33,235-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [1b9834d1] EVENT_ID: VM_NOT_RESPONDING(126), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM forum is not responding. 2017-10-01 15:59:33,235-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler7) [1b9834d1] VM 'ddf37076-e795-4c87-a4fb-9efcaae76d72'(dc1) moved from 'Paused' --> 'NotResponding' 2017-10-01 15:59:33,256-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [1b9834d1] EVENT_ID: VM_NOT_RESPONDING(126), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM dc1 is not responding. 2017-10-01 15:59:33,257-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler7) [1b9834d1] VM 'bfe9a524-2238-4bab-a91f-5dc6ed270896'(pcmode) moved from 'Paused' --> 'NotResponding' 2017-10-01 15:59:33,277-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [1b9834d1] EVENT_ID: VM_NOT_RESPONDING(126), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM pcmode is not responding. 2017-10-01 15:59:33,277-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler7) [1b9834d1] VM '1325cebf-834e-4571-a3e1-82beb0021ffe'(dns1) moved from 'Paused' --> 'NotResponding' 2017-10-01 15:59:33,298-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [1b9834d1] EVENT_ID: VM_NOT_RESPONDING(126), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM dns1 is not responding. 2017-10-01 15:59:33,298-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler7) [1b9834d1] VM 'a82b4cc3-6ef6-4a8e-97ef-1d4ea1b7e07c'(indigo) moved from 'Paused' --> 'NotResponding' 2017-10-01 15:59:33,319-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [1b9834d1] EVENT_ID: VM_NOT_RESPONDING(126), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM indigo is not responding. 2017-10-01 15:59:33,319-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler7) [1b9834d1] VM 'c8682859-9e55-460a-9ef0-d03e4a85f50e'(dhcp1) moved from 'Paused' --> 'NotResponding' 2017-10-01 15:59:33,340-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [1b9834d1] EVENT_ID: VM_NOT_RESPONDING(126), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM dhcp1 is not responding. 2017-10-01 15:59:33,340-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler7) [1b9834d1] VM '3bb83ad5-dc00-445d-8708-2db18c53e0e6'(www-vhost) moved from 'Paused' --> 'NotResponding' 2017-10-01 15:59:33,361-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [1b9834d1] EVENT_ID: VM_NOT_RESPONDING(126), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM www-vhost is not responding. 2017-10-01 15:59:33,361-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler7) [1b9834d1] VM 'a8d4e149-55ec-4961-a414-d7d1e1b05a59'(fix) moved from 'Paused' --> 'NotResponding' 2017-10-01 15:59:33,382-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [1b9834d1] EVENT_ID: VM_NOT_RESPONDING(126), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM fix is not responding. 2017-10-01 15:59:33,382-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler7) [1b9834d1] VM 'aa487207-7ff4-465a-9d9b-2a103d50dc77'(ftp1) moved from 'Paused' --> 'NotResponding' 2017-10-01 15:59:33,403-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [1b9834d1] EVENT_ID: VM_NOT_RESPONDING(126), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM ftp1 is not responding. 2017-10-01 15:59:33,403-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler7) [1b9834d1] VM '4ba80442-b361-4026-ae8c-895fd28042b0'(dist) moved from 'Paused' --> 'NotResponding' 2017-10-01 15:59:33,424-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [1b9834d1] EVENT_ID: VM_NOT_RESPONDING(126), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM dist is not responding. 2017-10-01 15:59:46,977-04 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-6-thread-25) [] Executing a command 'java.util.concurrent.FutureTask', but note that there are 9 tasks in the queue. 2017-10-01 15:59:46,977-04 INFO [org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner] (org.ovirt.thread.pool-6-thread-25) [2bf4b242-4939-4e61-976b-eda93f156af5] Start running Validate for command number 1/10 (Command type 'StopVm') 2017-10-01 15:59:46,977-04 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-6-thread-20) [] Executing a command 'java.util.concurrent.FutureTask', but note that there are 8 tasks in the queue. 2017-10-01 15:59:46,977-04 INFO [org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner] (org.ovirt.thread.pool-6-thread-20) [2bf4b242-4939-4e61-976b-eda93f156af5] Start running Validate for command number 2/10 (Command type 'StopVm') 2017-10-01 15:59:46,977-04 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-6-thread-15) [] Executing a command 'java.util.concurrent.FutureTask', but note that there are 7 tasks in the queue. 2017-10-01 15:59:46,977-04 INFO [org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner] (org.ovirt.thread.pool-6-thread-15) [2bf4b242-4939-4e61-976b-eda93f156af5] Start running Validate for command number 3/10 (Command type 'StopVm') 2017-10-01 15:59:46,977-04 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-6-thread-16) [] Executing a command 'java.util.concurrent.FutureTask', but note that there are 6 tasks in the queue. 2017-10-01 15:59:46,977-04 INFO [org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner] (org.ovirt.thread.pool-6-thread-16) [2bf4b242-4939-4e61-976b-eda93f156af5] Start running Validate for command number 4/10 (Command type 'StopVm') 2017-10-01 15:59:46,977-04 INFO [org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner] (org.ovirt.thread.pool-6-thread-10) [2bf4b242-4939-4e61-976b-eda93f156af5] Start running Validate for command number 5/10 (Command type 'StopVm') 2017-10-01 15:59:46,977-04 INFO [org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner] (org.ovirt.thread.pool-6-thread-27) [2bf4b242-4939-4e61-976b-eda93f156af5] Start running Validate for command number 6/10 (Command type 'StopVm') 2017-10-01 15:59:46,977-04 INFO [org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner] (org.ovirt.thread.pool-6-thread-1) [2bf4b242-4939-4e61-976b-eda93f156af5] Start running Validate for command number 7/10 (Command type 'StopVm') 2017-10-01 15:59:46,977-04 INFO [org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner] (org.ovirt.thread.pool-6-thread-19) [2bf4b242-4939-4e61-976b-eda93f156af5] Start running Validate for command number 8/10 (Command type 'StopVm') 2017-10-01 15:59:46,977-04 INFO [org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner] (org.ovirt.thread.pool-6-thread-42) [2bf4b242-4939-4e61-976b-eda93f156af5] Start running Validate for command number 10/10 (Command type 'StopVm') 2017-10-01 15:59:46,977-04 INFO [org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner] (org.ovirt.thread.pool-6-thread-39) [2bf4b242-4939-4e61-976b-eda93f156af5] Start running Validate for command number 9/10 (Command type 'StopVm') 2017-10-01 15:59:46,981-04 INFO [org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner] (org.ovirt.thread.pool-6-thread-25) [2bf4b242-4939-4e61-976b-eda93f156af5] Finish handling Validate for command number 1/10 (Command type 'StopVm') 2017-10-01 15:59:46,989-04 INFO [org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner] (org.ovirt.thread.pool-6-thread-15) [2bf4b242-4939-4e61-976b-eda93f156af5] Finish handling Validate for command number 3/10 (Command type 'StopVm') 2017-10-01 15:59:46,989-04 INFO [org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner] (org.ovirt.thread.pool-6-thread-19) [2bf4b242-4939-4e61-976b-eda93f156af5] Finish handling Validate for command number 8/10 (Command type 'StopVm') 2017-10-01 15:59:47,031-04 INFO [org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner] (org.ovirt.thread.pool-6-thread-27) [2bf4b242-4939-4e61-976b-eda93f156af5] Finish handling Validate for command number 6/10 (Command type 'StopVm') 2017-10-01 15:59:47,031-04 INFO [org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner] (org.ovirt.thread.pool-6-thread-10) [2bf4b242-4939-4e61-976b-eda93f156af5] Finish handling Validate for command number 5/10 (Command type 'StopVm') 2017-10-01 15:59:47,031-04 INFO [org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner] (org.ovirt.thread.pool-6-thread-16) [2bf4b242-4939-4e61-976b-eda93f156af5] Finish handling Validate for command number 4/10 (Command type 'StopVm') 2017-10-01 15:59:47,032-04 INFO [org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner] (org.ovirt.thread.pool-6-thread-20) [2bf4b242-4939-4e61-976b-eda93f156af5] Finish handling Validate for command number 2/10 (Command type 'StopVm') 2017-10-01 15:59:47,068-04 INFO [org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner] (org.ovirt.thread.pool-6-thread-39) [2bf4b242-4939-4e61-976b-eda93f156af5] Finish handling Validate for command number 9/10 (Command type 'StopVm') 2017-10-01 15:59:47,069-04 INFO [org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner] (org.ovirt.thread.pool-6-thread-1) [2bf4b242-4939-4e61-976b-eda93f156af5] Finish handling Validate for command number 7/10 (Command type 'StopVm') 2017-10-01 15:59:47,068-04 INFO [org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner] (org.ovirt.thread.pool-6-thread-42) [2bf4b242-4939-4e61-976b-eda93f156af5] Finish handling Validate for command number 10/10 (Command type 'StopVm') 2017-10-01 15:59:47,070-04 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-6-thread-22) [] Executing a command 'java.util.concurrent.FutureTask', but note that there are 9 tasks in the queue. 2017-10-01 15:59:47,070-04 INFO [org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner] (org.ovirt.thread.pool-6-thread-22) [2bf4b242-4939-4e61-976b-eda93f156af5] Start running Validate for command number 11/20 (Command type 'StopVm') 2017-10-01 15:59:47,070-04 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-6-thread-31) [] Executing a command 'java.util.concurrent.FutureTask', but note that there are 8 tasks in the queue. 2017-10-01 15:59:47,070-04 INFO [org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner] (org.ovirt.thread.pool-6-thread-31) [2bf4b242-4939-4e61-976b-eda93f156af5] Start running Validate for command number 12/20 (Command type 'StopVm') 2017-10-01 15:59:47,070-04 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-6-thread-45) [] Executing a command 'java.util.concurrent.FutureTask', but note that there are 7 tasks in the queue. 2017-10-01 15:59:47,070-04 INFO [org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner] (org.ovirt.thread.pool-6-thread-45) [2bf4b242-4939-4e61-976b-eda93f156af5] Start running Validate for command number 13/20 (Command type 'StopVm') 2017-10-01 15:59:47,070-04 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-6-thread-47) [] Executing a command 'java.util.concurrent.FutureTask', but note that there are 6 tasks in the queue. 2017-10-01 15:59:47,070-04 INFO [org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner] (org.ovirt.thread.pool-6-thread-47) [2bf4b242-4939-4e61-976b-eda93f156af5] Start running Validate for command number 14/20 (Command type 'StopVm') 2017-10-01 15:59:47,070-04 INFO [org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner] (org.ovirt.thread.pool-6-thread-3) [2bf4b242-4939-4e61-976b-eda93f156af5] Start running Validate for command number 15/20 (Command type 'StopVm') 2017-10-01 15:59:47,070-04 INFO [org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner] (org.ovirt.thread.pool-6-thread-12) [2bf4b242-4939-4e61-976b-eda93f156af5] Start running Validate for command number 16/20 (Command type 'StopVm') 2017-10-01 15:59:47,070-04 INFO [org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner] (org.ovirt.thread.pool-6-thread-17) [2bf4b242-4939-4e61-976b-eda93f156af5] Start running Validate for command number 17/20 (Command type 'StopVm') 2017-10-01 15:59:47,070-04 INFO [org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner] (org.ovirt.thread.pool-6-thread-30) [2bf4b242-4939-4e61-976b-eda93f156af5] Start running Validate for command number 18/20 (Command type 'StopVm') 2017-10-01 15:59:47,070-04 INFO [org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner] (org.ovirt.thread.pool-6-thread-37) [2bf4b242-4939-4e61-976b-eda93f156af5] Start running Validate for command number 19/20 (Command type 'StopVm') 2017-10-01 15:59:47,070-04 INFO [org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner] (org.ovirt.thread.pool-6-thread-43) [2bf4b242-4939-4e61-976b-eda93f156af5] Start running Validate for command number 20/20 (Command type 'StopVm') 2017-10-01 15:59:47,074-04 INFO [org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner] (org.ovirt.thread.pool-6-thread-22) [2bf4b242-4939-4e61-976b-eda93f156af5] Finish handling Validate for command number 11/20 (Command type 'StopVm') 2017-10-01 15:59:47,074-04 INFO [org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner] (org.ovirt.thread.pool-6-thread-17) [2bf4b242-4939-4e61-976b-eda93f156af5] Finish handling Validate for command number 17/20 (Command type 'StopVm') 2017-10-01 15:59:47,075-04 INFO [org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner] (org.ovirt.thread.pool-6-thread-31) [2bf4b242-4939-4e61-976b-eda93f156af5] Finish handling Validate for command number 12/20 (Command type 'StopVm') 2017-10-01 15:59:47,075-04 INFO [org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner] (org.ovirt.thread.pool-6-thread-12) [2bf4b242-4939-4e61-976b-eda93f156af5] Finish handling Validate for command number 16/20 (Command type 'StopVm') 2017-10-01 15:59:47,075-04 INFO [org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner] (org.ovirt.thread.pool-6-thread-43) [2bf4b242-4939-4e61-976b-eda93f156af5] Finish handling Validate for command number 20/20 (Command type 'StopVm') 2017-10-01 15:59:47,075-04 INFO [org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner] (org.ovirt.thread.pool-6-thread-37) [2bf4b242-4939-4e61-976b-eda93f156af5] Finish handling Validate for command number 19/20 (Command type 'StopVm') 2017-10-01 15:59:47,075-04 INFO [org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner] (org.ovirt.thread.pool-6-thread-30) [2bf4b242-4939-4e61-976b-eda93f156af5] Finish handling Validate for command number 18/20 (Command type 'StopVm') 2017-10-01 15:59:47,122-04 INFO [org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner] (org.ovirt.thread.pool-6-thread-3) [2bf4b242-4939-4e61-976b-eda93f156af5] Finish handling Validate for command number 15/20 (Command type 'StopVm') 2017-10-01 15:59:47,122-04 INFO [org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner] (org.ovirt.thread.pool-6-thread-45) [2bf4b242-4939-4e61-976b-eda93f156af5] Finish handling Validate for command number 13/20 (Command type 'StopVm') 2017-10-01 15:59:47,126-04 INFO [org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner] (org.ovirt.thread.pool-6-thread-47) [2bf4b242-4939-4e61-976b-eda93f156af5] Finish handling Validate for command number 14/20 (Command type 'StopVm') 2017-10-01 15:59:47,126-04 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-6-thread-14) [] Executing a command 'java.util.concurrent.FutureTask', but note that there are 9 tasks in the queue. 2017-10-01 15:59:47,126-04 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-6-thread-4) [] Executing a command 'java.util.concurrent.FutureTask', but note that there are 8 tasks in the queue. 2017-10-01 15:59:47,126-04 INFO [org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner] (org.ovirt.thread.pool-6-thread-14) [2bf4b242-4939-4e61-976b-eda93f156af5] Start running Validate for command number 21/30 (Command type 'StopVm') 2017-10-01 15:59:47,126-04 INFO [org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner] (org.ovirt.thread.pool-6-thread-4) [2bf4b242-4939-4e61-976b-eda93f156af5] Start running Validate for command number 22/30 (Command type 'StopVm') 2017-10-01 15:59:47,126-04 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-6-thread-18) [] Executing a command 'java.util.concurrent.FutureTask', but note that there are 7 tasks in the queue. 2017-10-01 15:59:47,126-04 INFO [org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner] (org.ovirt.thread.pool-6-thread-48) [2bf4b242-4939-4e61-976b-eda93f156af5] Start running Validate for command number 25/30 (Command type 'StopVm') 2017-10-01 15:59:47,126-04 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-6-thread-50) [] Executing a command 'java.util.concurrent.FutureTask', but note that there are 6 tasks in the queue. 2017-10-01 15:59:47,126-04 INFO [org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner] (org.ovirt.thread.pool-6-thread-18) [2bf4b242-4939-4e61-976b-eda93f156af5] Start running Validate for command number 23/30 (Command type 'StopVm') 2017-10-01 15:59:47,126-04 INFO [org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner] (org.ovirt.thread.pool-6-thread-6) [2bf4b242-4939-4e61-976b-eda93f156af5] Start running Validate for command number 26/30 (Command type 'StopVm') 2017-10-01 15:59:47,126-04 INFO [org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner] (org.ovirt.thread.pool-6-thread-7) [2bf4b242-4939-4e61-976b-eda93f156af5] Start running Validate for command number 27/30 (Command type 'StopVm') 2017-10-01 15:59:47,126-04 INFO [org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner] (org.ovirt.thread.pool-6-thread-50) [2bf4b242-4939-4e61-976b-eda93f156af5] Start running Validate for command number 24/30 (Command type 'StopVm') 2017-10-01 15:59:47,126-04 INFO [org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner] (org.ovirt.thread.pool-6-thread-21) [2bf4b242-4939-4e61-976b-eda93f156af5] Start running Validate for command number 28/30 (Command type 'StopVm') 2017-10-01 15:59:47,127-04 INFO [org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner] (org.ovirt.thread.pool-6-thread-36) [2bf4b242-4939-4e61-976b-eda93f156af5] Start running Validate for command number 29/30 (Command type 'StopVm') 2017-10-01 15:59:47,127-04 INFO [org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner] (org.ovirt.thread.pool-6-thread-35) [2bf4b242-4939-4e61-976b-eda93f156af5] Start running Validate for command number 30/30 (Command type 'StopVm') 2017-10-01 15:59:47,130-04 INFO [org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner] (org.ovirt.thread.pool-6-thread-21) [2bf4b242-4939-4e61-976b-eda93f156af5] Finish handling Validate for command number 28/30 (Command type 'StopVm') 2017-10-01 15:59:47,130-04 INFO [org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner] (org.ovirt.thread.pool-6-thread-6) [2bf4b242-4939-4e61-976b-eda93f156af5] Finish handling Validate for command number 26/30 (Command type 'StopVm') 2017-10-01 15:59:47,130-04 INFO [org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner] (org.ovirt.thread.pool-6-thread-4) [2bf4b242-4939-4e61-976b-eda93f156af5] Finish handling Validate for command number 22/30 (Command type 'StopVm') 2017-10-01 15:59:47,131-04 INFO [org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner] (org.ovirt.thread.pool-6-thread-7) [2bf4b242-4939-4e61-976b-eda93f156af5] Finish handling Validate for command number 27/30 (Command type 'StopVm') 2017-10-01 15:59:47,131-04 INFO [org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner] (org.ovirt.thread.pool-6-thread-50) [2bf4b242-4939-4e61-976b-eda93f156af5] Finish handling Validate for command number 24/30 (Command type 'StopVm') 2017-10-01 15:59:47,131-04 INFO [org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner] (org.ovirt.thread.pool-6-thread-35) [2bf4b242-4939-4e61-976b-eda93f156af5] Finish handling Validate for command number 30/30 (Command type 'StopVm') 2017-10-01 15:59:47,131-04 INFO [org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner] (org.ovirt.thread.pool-6-thread-36) [2bf4b242-4939-4e61-976b-eda93f156af5] Finish handling Validate for command number 29/30 (Command type 'StopVm') 2017-10-01 15:59:47,174-04 INFO [org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner] (org.ovirt.thread.pool-6-thread-18) [2bf4b242-4939-4e61-976b-eda93f156af5] Finish handling Validate for command number 23/30 (Command type 'StopVm') 2017-10-01 15:59:47,175-04 INFO [org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner] (org.ovirt.thread.pool-6-thread-48) [2bf4b242-4939-4e61-976b-eda93f156af5] Finish handling Validate for command number 25/30 (Command type 'StopVm') 2017-10-01 15:59:47,178-04 INFO [org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner] (org.ovirt.thread.pool-6-thread-14) [2bf4b242-4939-4e61-976b-eda93f156af5] Finish handling Validate for command number 21/30 (Command type 'StopVm') 2017-10-01 15:59:47,178-04 INFO [org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner] (org.ovirt.thread.pool-6-thread-41) [2bf4b242-4939-4e61-976b-eda93f156af5] Start running Validate for command number 31/35 (Command type 'StopVm') 2017-10-01 15:59:47,178-04 INFO [org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner] (org.ovirt.thread.pool-6-thread-24) [2bf4b242-4939-4e61-976b-eda93f156af5] Start running Validate for command number 32/35 (Command type 'StopVm') 2017-10-01 15:59:47,178-04 INFO [org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner] (org.ovirt.thread.pool-6-thread-2) [2bf4b242-4939-4e61-976b-eda93f156af5] Start running Validate for command number 33/35 (Command type 'StopVm') 2017-10-01 15:59:47,178-04 INFO [org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner] (org.ovirt.thread.pool-6-thread-40) [2bf4b242-4939-4e61-976b-eda93f156af5] Start running Validate for command number 34/35 (Command type 'StopVm') 2017-10-01 15:59:47,178-04 INFO [org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner] (org.ovirt.thread.pool-6-thread-49) [2bf4b242-4939-4e61-976b-eda93f156af5] Start running Validate for command number 35/35 (Command type 'StopVm') 2017-10-01 15:59:47,182-04 INFO [org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner] (org.ovirt.thread.pool-6-thread-49) [2bf4b242-4939-4e61-976b-eda93f156af5] Finish handling Validate for command number 35/35 (Command type 'StopVm') 2017-10-01 15:59:47,182-04 INFO [org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner] (org.ovirt.thread.pool-6-thread-2) [2bf4b242-4939-4e61-976b-eda93f156af5] Finish handling Validate for command number 33/35 (Command type 'StopVm') 2017-10-01 15:59:47,225-04 INFO [org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner] (org.ovirt.thread.pool-6-thread-41) [2bf4b242-4939-4e61-976b-eda93f156af5] Finish handling Validate for command number 31/35 (Command type 'StopVm') 2017-10-01 15:59:47,226-04 INFO [org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner] (org.ovirt.thread.pool-6-thread-40) [2bf4b242-4939-4e61-976b-eda93f156af5] Finish handling Validate for command number 34/35 (Command type 'StopVm') 2017-10-01 15:59:47,226-04 INFO [org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner] (org.ovirt.thread.pool-6-thread-24) [2bf4b242-4939-4e61-976b-eda93f156af5] Finish handling Validate for command number 32/35 (Command type 'StopVm') 2017-10-01 15:59:47,241-04 INFO [org.ovirt.engine.core.bll.StopVmCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] Running command: StopVmCommand internal: false. Entities affected : ID: bcd5776e-978b-4723-933d-70a80322a796 Type: VMAction group STOP_VM with role type USER 2017-10-01 15:59:47,284-04 INFO [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] START, DestroyVmVDSCommand( DestroyVmVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vmId='bcd5776e-978b-4723-933d-70a80322a796', force='false', secondsToWait='0', gracefully='false', reason='', ignoreNoVm='false'}), log id: 69ffed24 2017-10-01 15:59:47,284-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] START, DestroyVDSCommand(HostName = virt4, DestroyVmVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vmId='bcd5776e-978b-4723-933d-70a80322a796', force='false', secondsToWait='0', gracefully='false', reason='', ignoreNoVm='false'}), log id: 52974e96 2017-10-01 15:59:56,193-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [293f56a8] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 16:00:00,015-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [1972f041] Autorecovering 3 storage domains 2017-10-01 16:00:00,015-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [1972f041] Autorecovering storage domains id: 63d680c7-4c94-4b29-a6cb-f288a128b9bc 2017-10-01 16:00:00,019-04 INFO [org.ovirt.engine.core.bll.storage.connection.ConnectDomainToStorageCommand] (DefaultQuartzScheduler6) [535c1f37] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: 63d680c7-4c94-4b29-a6cb-f288a128b9bc Type: Storage 2017-10-01 16:00:00,019-04 INFO [org.ovirt.engine.core.bll.storage.connection.ConnectDomainToStorageCommand] (DefaultQuartzScheduler6) [535c1f37] ConnectDomainToStorage. Before Connect all hosts to pool. Time: Sun Oct 01 16:00:00 EDT 2017 2017-10-01 16:00:00,095-04 INFO [org.ovirt.engine.core.bll.storage.connection.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-6-thread-34) [36e9b1db] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN 2017-10-01 16:00:00,095-04 INFO [org.ovirt.engine.core.bll.storage.connection.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-6-thread-33) [12f23e8d] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN 2017-10-01 16:00:00,095-04 INFO [org.ovirt.engine.core.bll.storage.connection.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-6-thread-38) [2b4b11a] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN 2017-10-01 16:00:00,101-04 INFO [org.ovirt.engine.core.bll.storage.connection.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-6-thread-23) [6d81b0a3] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN 2017-10-01 16:00:00,112-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-33) [12f23e8d] START, ConnectStorageServerVDSCommand(HostName = virt1, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='00000000-0000-0000-0000-000000000000', storageType='NFS', connectionList='[StorageServerConnections:{id='13cf1e67-9961-4a25-a41f-cc76b4ab793d', connection='virtfs-fs:/nfs/iso', iqn='null', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 4aa48e45 2017-10-01 16:00:00,126-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-23) [6d81b0a3] START, ConnectStorageServerVDSCommand(HostName = virt4, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='00000000-0000-0000-0000-000000000000', storageType='NFS', connectionList='[StorageServerConnections:{id='13cf1e67-9961-4a25-a41f-cc76b4ab793d', connection='virtfs-fs:/nfs/iso', iqn='null', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 13d9ef27 2017-10-01 16:00:00,134-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-38) [2b4b11a] START, ConnectStorageServerVDSCommand(HostName = virt2, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='00000000-0000-0000-0000-000000000000', storageType='NFS', connectionList='[StorageServerConnections:{id='13cf1e67-9961-4a25-a41f-cc76b4ab793d', connection='virtfs-fs:/nfs/iso', iqn='null', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 4fe75ba1 2017-10-01 16:00:00,141-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-34) [36e9b1db] START, ConnectStorageServerVDSCommand(HostName = virt3, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='00000000-0000-0000-0000-000000000000', storageType='NFS', connectionList='[StorageServerConnections:{id='13cf1e67-9961-4a25-a41f-cc76b4ab793d', connection='virtfs-fs:/nfs/iso', iqn='null', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: fe12809 2017-10-01 16:00:03,481-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler5) [23c74a4] VM 'db3e08fb-c111-442d-aeec-1507da30f486'(print) moved from 'NotResponding' --> 'Paused' 2017-10-01 16:00:03,503-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [23c74a4] EVENT_ID: VM_PAUSED(1,025), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM print has been paused. 2017-10-01 16:00:03,524-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [23c74a4] EVENT_ID: VM_PAUSED_ERROR(139), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM print has been paused due to unknown storage error. 2017-10-01 16:00:03,525-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler5) [23c74a4] VM 'ae56c31c-71f7-433a-b198-a1c2c846325e'(forum) moved from 'NotResponding' --> 'Paused' 2017-10-01 16:00:03,545-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [23c74a4] EVENT_ID: VM_PAUSED(1,025), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM forum has been paused. 2017-10-01 16:00:03,566-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [23c74a4] EVENT_ID: VM_PAUSED_ERROR(139), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM forum has been paused due to unknown storage error. 2017-10-01 16:00:03,567-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler5) [23c74a4] VM 'ddf37076-e795-4c87-a4fb-9efcaae76d72'(dc1) moved from 'NotResponding' --> 'Paused' 2017-10-01 16:00:03,588-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [23c74a4] EVENT_ID: VM_PAUSED(1,025), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM dc1 has been paused. 2017-10-01 16:00:03,609-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [23c74a4] EVENT_ID: VM_PAUSED_ERROR(139), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM dc1 has been paused due to unknown storage error. 2017-10-01 16:00:03,609-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler5) [23c74a4] VM 'bfe9a524-2238-4bab-a91f-5dc6ed270896'(pcmode) moved from 'NotResponding' --> 'Paused' 2017-10-01 16:00:03,630-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [23c74a4] EVENT_ID: VM_PAUSED(1,025), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM pcmode has been paused. 2017-10-01 16:00:03,650-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [23c74a4] EVENT_ID: VM_PAUSED_ERROR(139), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM pcmode has been paused due to unknown storage error. 2017-10-01 16:00:03,651-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler5) [23c74a4] VM '1325cebf-834e-4571-a3e1-82beb0021ffe'(dns1) moved from 'NotResponding' --> 'Paused' 2017-10-01 16:00:03,672-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [23c74a4] EVENT_ID: VM_PAUSED(1,025), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM dns1 has been paused. 2017-10-01 16:00:03,693-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [23c74a4] EVENT_ID: VM_PAUSED_ERROR(139), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM dns1 has been paused due to unknown storage error. 2017-10-01 16:00:03,693-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler5) [23c74a4] VM 'a82b4cc3-6ef6-4a8e-97ef-1d4ea1b7e07c'(indigo) moved from 'NotResponding' --> 'Paused' 2017-10-01 16:00:03,714-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [23c74a4] EVENT_ID: VM_PAUSED(1,025), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM indigo has been paused. 2017-10-01 16:00:03,735-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [23c74a4] EVENT_ID: VM_PAUSED_ERROR(139), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM indigo has been paused due to unknown storage error. 2017-10-01 16:00:03,735-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler5) [23c74a4] VM 'c8682859-9e55-460a-9ef0-d03e4a85f50e'(dhcp1) moved from 'NotResponding' --> 'Paused' 2017-10-01 16:00:03,756-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [23c74a4] EVENT_ID: VM_PAUSED(1,025), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM dhcp1 has been paused. 2017-10-01 16:00:03,776-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [23c74a4] EVENT_ID: VM_PAUSED_ERROR(139), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM dhcp1 has been paused due to unknown storage error. 2017-10-01 16:00:03,777-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler5) [23c74a4] VM '3bb83ad5-dc00-445d-8708-2db18c53e0e6'(www-vhost) moved from 'NotResponding' --> 'Paused' 2017-10-01 16:00:03,798-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [23c74a4] EVENT_ID: VM_PAUSED(1,025), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM www-vhost has been paused. 2017-10-01 16:00:03,818-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [23c74a4] EVENT_ID: VM_PAUSED_ERROR(139), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM www-vhost has been paused due to unknown storage error. 2017-10-01 16:00:03,819-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler5) [23c74a4] VM 'a8d4e149-55ec-4961-a414-d7d1e1b05a59'(fix) moved from 'NotResponding' --> 'Paused' 2017-10-01 16:00:03,839-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [23c74a4] EVENT_ID: VM_PAUSED(1,025), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM fix has been paused. 2017-10-01 16:00:03,860-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [23c74a4] EVENT_ID: VM_PAUSED_ERROR(139), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM fix has been paused due to unknown storage error. 2017-10-01 16:00:03,860-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler5) [23c74a4] VM 'aa487207-7ff4-465a-9d9b-2a103d50dc77'(ftp1) moved from 'NotResponding' --> 'Paused' 2017-10-01 16:00:03,881-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [23c74a4] EVENT_ID: VM_PAUSED(1,025), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM ftp1 has been paused. 2017-10-01 16:00:03,901-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [23c74a4] EVENT_ID: VM_PAUSED_ERROR(139), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM ftp1 has been paused due to unknown storage error. 2017-10-01 16:00:03,902-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler5) [23c74a4] VM '4ba80442-b361-4026-ae8c-895fd28042b0'(dist) moved from 'NotResponding' --> 'Paused' 2017-10-01 16:00:03,922-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [23c74a4] EVENT_ID: VM_PAUSED(1,025), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM dist has been paused. 2017-10-01 16:00:03,943-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [23c74a4] EVENT_ID: VM_PAUSED_ERROR(139), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM dist has been paused due to unknown storage error. 2017-10-01 16:00:16,349-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler9) [30b477be] VM '07331c78-6e09-41b4-9b29-edda32f46351'(samba) moved from 'Paused' --> 'NotResponding' 2017-10-01 16:00:16,371-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [30b477be] EVENT_ID: VM_NOT_RESPONDING(126), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM samba is not responding. 2017-10-01 16:00:16,372-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler9) [30b477be] VM 'dc66c051-24b7-4cb8-99eb-db398d396ef1'(samba1) moved from 'Paused' --> 'NotResponding' 2017-10-01 16:00:16,393-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [30b477be] EVENT_ID: VM_NOT_RESPONDING(126), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM samba1 is not responding. 2017-10-01 16:00:16,394-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler9) [30b477be] VM '3e5a5728-8940-4fe7-b4af-11fb8e12f116'(www) moved from 'Paused' --> 'NotResponding' 2017-10-01 16:00:16,415-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [30b477be] EVENT_ID: VM_NOT_RESPONDING(126), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM www is not responding. 2017-10-01 16:00:16,415-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler9) [30b477be] VM 'f87ec08d-0fdf-42ba-8aac-c66c404d7aae'(webapp) moved from 'Paused' --> 'NotResponding' 2017-10-01 16:00:16,436-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [30b477be] EVENT_ID: VM_NOT_RESPONDING(126), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM webapp is not responding. 2017-10-01 16:00:16,436-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler9) [30b477be] VM '911d6883-607c-464e-b46b-8e6141bdd2de'(hdb) moved from 'Paused' --> 'NotResponding' 2017-10-01 16:00:16,457-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [30b477be] EVENT_ID: VM_NOT_RESPONDING(126), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM hdb is not responding. 2017-10-01 16:00:16,457-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler9) [30b477be] VM '8391f9a7-7fe7-4025-8fba-6273ae587a52'(archive) moved from 'Paused' --> 'NotResponding' 2017-10-01 16:00:16,478-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [30b477be] EVENT_ID: VM_NOT_RESPONDING(126), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM archive is not responding. 2017-10-01 16:00:16,478-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler9) [30b477be] VM '1c7e966f-29e8-4788-b64c-86e52f415061'(udb) moved from 'Paused' --> 'NotResponding' 2017-10-01 16:00:16,498-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [30b477be] EVENT_ID: VM_NOT_RESPONDING(126), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM udb is not responding. 2017-10-01 16:00:16,498-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler9) [30b477be] VM '9ea2821e-557f-45cb-9872-1e6c5aeaf246'(rs) moved from 'Paused' --> 'NotResponding' 2017-10-01 16:00:16,519-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [30b477be] EVENT_ID: VM_NOT_RESPONDING(126), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM rs is not responding. 2017-10-01 16:00:16,519-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler9) [30b477be] VM 'd66235b4-7c8d-4da2-a572-82ecc9c53a59'(www-las) moved from 'Paused' --> 'NotResponding' 2017-10-01 16:00:16,539-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [30b477be] EVENT_ID: VM_NOT_RESPONDING(126), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM www-las is not responding. 2017-10-01 16:00:16,540-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler9) [30b477be] VM '5885c43a-b133-45dc-b8d5-1aa2a4a19579'(prs) moved from 'Up' --> 'NotResponding' 2017-10-01 16:00:16,560-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [30b477be] EVENT_ID: VM_NOT_RESPONDING(126), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM prs is not responding. 2017-10-01 16:00:16,560-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler9) [30b477be] VM 'ae65a104-ed3f-40e0-ba92-5708cf338918'(vpn2) moved from 'Paused' --> 'NotResponding' 2017-10-01 16:00:16,581-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [30b477be] EVENT_ID: VM_NOT_RESPONDING(126), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM vpn2 is not responding. 2017-10-01 16:00:16,581-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler9) [30b477be] VM 'd56363ad-9cb1-4a70-a33a-ce439b044d7d'(svn) moved from 'Paused' --> 'NotResponding' 2017-10-01 16:00:16,601-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [30b477be] EVENT_ID: VM_NOT_RESPONDING(126), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM svn is not responding. 2017-10-01 16:00:46,454-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [151ed11b] FINISH, ConnectStoragePoolVDSCommand, log id: 3ef1998 2017-10-01 16:00:46,454-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler2) [151ed11b] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNonOperationalException: Could not connect host to Data Center(Storage issue) 2017-10-01 16:00:46,493-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-38) [2b4b11a] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command ConnectStorageServerVDS failed: Vds timeout occured 2017-10-01 16:00:46,493-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-38) [2b4b11a] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand' return value ' 2017-10-01 16:00:46,493-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-38) [2b4b11a] HostName = virt2 2017-10-01 16:00:46,494-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-38) [2b4b11a] Command 'ConnectStorageServerVDSCommand(HostName = virt2, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='00000000-0000-0000-0000-000000000000', storageType='NFS', connectionList='[StorageServerConnections:{id='13cf1e67-9961-4a25-a41f-cc76b4ab793d', connection='virtfs-fs:/nfs/iso', iqn='null', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'})' execution failed: VDSGenericException: VDSNetworkException: Vds timeout occured 2017-10-01 16:00:46,494-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-38) [2b4b11a] FINISH, ConnectStorageServerVDSCommand, log id: 4fe75ba1 2017-10-01 16:00:46,494-04 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (org.ovirt.thread.pool-6-thread-7) [2b4b11a] Host 'virt2' is not responding. It will stay in Connecting state for a grace period of 65 seconds and after that an attempt to fence the host will be issued. 2017-10-01 16:00:46,494-04 ERROR [org.ovirt.engine.core.bll.storage.connection.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-6-thread-38) [2b4b11a] Command 'org.ovirt.engine.core.bll.storage.connection.ConnectStorageToVdsCommand' failed: EngineException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Vds timeout occured (Failed with error VDS_NETWORK_ERROR and code 5022) 2017-10-01 16:00:46,504-04 ERROR [org.ovirt.engine.core.bll.storage.connection.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-6-thread-38) [2b4b11a] Transaction rolled-back for command 'org.ovirt.engine.core.bll.storage.connection.ConnectStorageToVdsCommand'. 2017-10-01 16:00:46,537-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-7) [2b4b11a] EVENT_ID: VDS_HOST_NOT_RESPONDING_CONNECTING(9,008), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host virt2 is not responding. It will stay in Connecting state for a grace period of 65 seconds and after that an attempt to fence the host will be issued. 2017-10-01 16:00:49,008-04 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to virt2-mgmt/192.168.0.35 2017-10-01 16:00:50,239-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [7f598ea] START, GetHardwareInfoVDSCommand(HostName = virt2, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vds='Host[virt2,a0ccfd11-3402-4eba-a119-4987561b738e]'}), log id: 258ea583 2017-10-01 16:00:51,243-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler8) [7f598ea] FINISH, GetHardwareInfoVDSCommand, log id: 258ea583 2017-10-01 16:00:51,337-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler8) [28c6b5c9] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: a0ccfd11-3402-4eba-a119-4987561b738e Type: VDS 2017-10-01 16:00:51,362-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler8) [3b19cfcf] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: aca61eac-ed8a-484b-bbee-3645148f6405 Type: StoragePool 2017-10-01 16:00:51,368-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [2dfdaa98] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: aca61eac-ed8a-484b-bbee-3645148f6405 Type: StoragePool 2017-10-01 16:00:51,437-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [2dfdaa98] START, ConnectStorageServerVDSCommand(HostName = virt2, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', storageType='NFS', connectionList='[StorageServerConnections:{id='5e557659-6817-40fa-9a55-7458125404a6', connection='virtfs-fs:/nfs/data', iqn='null', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='13cf1e67-9961-4a25-a41f-cc76b4ab793d', connection='virtfs-fs:/nfs/iso', iqn='null', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='417fdd51-dbcd-41d9-9a05-ef4d9f485d13', connection='virtfs-fs:/nfs/export', iqn='null', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 2873e8 2017-10-01 16:00:56,213-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [3f940737] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 16:01:00,155-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-33) [12f23e8d] FINISH, ConnectStorageServerVDSCommand, return: {13cf1e67-9961-4a25-a41f-cc76b4ab793d=0}, log id: 4aa48e45 2017-10-01 16:01:00,285-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-34) [36e9b1db] FINISH, ConnectStorageServerVDSCommand, return: {13cf1e67-9961-4a25-a41f-cc76b4ab793d=0}, log id: fe12809 2017-10-01 16:01:05,108-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler4) [4127f80e] VM 'db3e08fb-c111-442d-aeec-1507da30f486'(print) moved from 'Paused' --> 'NotResponding' 2017-10-01 16:01:05,178-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [4127f80e] EVENT_ID: VM_NOT_RESPONDING(126), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM print is not responding. 2017-10-01 16:01:05,179-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler4) [4127f80e] VM 'ae56c31c-71f7-433a-b198-a1c2c846325e'(forum) moved from 'Paused' --> 'NotResponding' 2017-10-01 16:01:05,255-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [4127f80e] EVENT_ID: VM_NOT_RESPONDING(126), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM forum is not responding. 2017-10-01 16:01:05,255-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler4) [4127f80e] VM 'ddf37076-e795-4c87-a4fb-9efcaae76d72'(dc1) moved from 'Paused' --> 'NotResponding' 2017-10-01 16:01:05,279-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [4127f80e] EVENT_ID: VM_NOT_RESPONDING(126), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM dc1 is not responding. 2017-10-01 16:01:05,280-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler4) [4127f80e] VM 'bfe9a524-2238-4bab-a91f-5dc6ed270896'(pcmode) moved from 'Paused' --> 'NotResponding' 2017-10-01 16:01:05,303-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [4127f80e] EVENT_ID: VM_NOT_RESPONDING(126), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM pcmode is not responding. 2017-10-01 16:01:05,303-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler4) [4127f80e] VM '1325cebf-834e-4571-a3e1-82beb0021ffe'(dns1) moved from 'Paused' --> 'NotResponding' 2017-10-01 16:01:05,327-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [4127f80e] EVENT_ID: VM_NOT_RESPONDING(126), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM dns1 is not responding. 2017-10-01 16:01:05,327-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler4) [4127f80e] VM 'a82b4cc3-6ef6-4a8e-97ef-1d4ea1b7e07c'(indigo) moved from 'Paused' --> 'NotResponding' 2017-10-01 16:01:05,352-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [4127f80e] EVENT_ID: VM_NOT_RESPONDING(126), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM indigo is not responding. 2017-10-01 16:01:05,353-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler4) [4127f80e] VM 'c8682859-9e55-460a-9ef0-d03e4a85f50e'(dhcp1) moved from 'Paused' --> 'NotResponding' 2017-10-01 16:01:05,374-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [4127f80e] EVENT_ID: VM_NOT_RESPONDING(126), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM dhcp1 is not responding. 2017-10-01 16:01:05,374-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler4) [4127f80e] VM '3bb83ad5-dc00-445d-8708-2db18c53e0e6'(www-vhost) moved from 'Paused' --> 'NotResponding' 2017-10-01 16:01:05,395-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [4127f80e] EVENT_ID: VM_NOT_RESPONDING(126), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM www-vhost is not responding. 2017-10-01 16:01:05,396-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler4) [4127f80e] VM 'a8d4e149-55ec-4961-a414-d7d1e1b05a59'(fix) moved from 'Paused' --> 'NotResponding' 2017-10-01 16:01:05,417-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [4127f80e] EVENT_ID: VM_NOT_RESPONDING(126), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM fix is not responding. 2017-10-01 16:01:05,417-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler4) [4127f80e] VM 'aa487207-7ff4-465a-9d9b-2a103d50dc77'(ftp1) moved from 'Paused' --> 'NotResponding' 2017-10-01 16:01:05,438-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [4127f80e] EVENT_ID: VM_NOT_RESPONDING(126), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM ftp1 is not responding. 2017-10-01 16:01:05,438-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler4) [4127f80e] VM '4ba80442-b361-4026-ae8c-895fd28042b0'(dist) moved from 'Paused' --> 'NotResponding' 2017-10-01 16:01:05,459-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [4127f80e] EVENT_ID: VM_NOT_RESPONDING(126), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM dist is not responding. 2017-10-01 16:01:32,733-04 ERROR [org.ovirt.vdsm.jsonrpc.client.JsonRpcClient] (ResponseWorker) [] Not able to update response for "32fd90bd-af91-4b91-9c49-92e4e51a0e17" 2017-10-01 16:01:33,672-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [2dfdaa98] FINISH, ConnectStorageServerVDSCommand, return: {5e557659-6817-40fa-9a55-7458125404a6=0, 417fdd51-dbcd-41d9-9a05-ef4d9f485d13=0, 13cf1e67-9961-4a25-a41f-cc76b4ab793d=0}, log id: 2873e8 2017-10-01 16:01:33,692-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler8) [2dfdaa98] Host 'virt2' storage connection was succeeded 2017-10-01 16:01:33,702-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] START, ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 29d7b4ea 2017-10-01 16:01:33,704-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:01:33,795-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-6-thread-13) [f6c99088-149b-457e-a985-b2d5885c8a28] FINISH, DestroyVDSCommand, log id: 4788a20c 2017-10-01 16:01:33,796-04 INFO [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (org.ovirt.thread.pool-6-thread-13) [f6c99088-149b-457e-a985-b2d5885c8a28] FINISH, DestroyVmVDSCommand, log id: 5179072d 2017-10-01 16:01:33,796-04 ERROR [org.ovirt.vdsm.jsonrpc.client.JsonRpcClient] (ResponseWorker) [] Not able to update response for "d947bb1d-8055-43dd-957b-99fb69e4ffc6" 2017-10-01 16:01:33,801-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [4e22b08b] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:01:33,801-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler7) [4e22b08b] Command 'SpmStatusVDSCommand(HostName = virt2, SpmStatusVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:01:33,802-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] Destroy VM couldn't find VM 'bcd5776e-978b-4723-933d-70a80322a796'. If after Cancel Migration and VM is UP on source, ignore next error, it's just a clean-up call 2017-10-01 16:01:33,802-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] Failed in 'DestroyVDS' method 2017-10-01 16:01:33,805-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-13) [f6c99088-149b-457e-a985-b2d5885c8a28] EVENT_ID: USER_STOP_VM(33), Correlation ID: f6c99088-149b-457e-a985-b2d5885c8a28, Job ID: a3e49e74-ac66-4027-aa9b-a97a885ca635, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM cape powered off by admin@internal (Host: virt4). 2017-10-01 16:01:33,806-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command DestroyVDS failed: Virtual machine does not exist 2017-10-01 16:01:33,806-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand' return value 'StatusOnlyReturn [status=Status [code=1, message=Virtual machine does not exist]]' 2017-10-01 16:01:33,806-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] HostName = virt4 2017-10-01 16:01:33,806-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] Command 'DestroyVDSCommand(HostName = virt4, DestroyVmVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vmId='bcd5776e-978b-4723-933d-70a80322a796', force='false', secondsToWait='0', gracefully='false', reason='', ignoreNoVm='false'})' execution failed: VDSGenericException: VDSErrorException: Failed to DestroyVDS, error = Virtual machine does not exist, code = 1 2017-10-01 16:01:33,806-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] FINISH, DestroyVDSCommand, log id: 52974e96 2017-10-01 16:01:33,806-04 ERROR [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] Failed to destroy VM 'bcd5776e-978b-4723-933d-70a80322a796' in VDS = '5ae29ed2-9c0f-4754-8df3-b43d14726846' , error = 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to DestroyVDS, error = Virtual machine does not exist, code = 1' 2017-10-01 16:01:33,806-04 INFO [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] FINISH, DestroyVmVDSCommand, log id: 69ffed24 2017-10-01 16:01:33,806-04 ERROR [org.ovirt.engine.core.bll.StopVmCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] Command 'org.ovirt.engine.core.bll.StopVmCommand' failed: EngineException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to DestroyVDS, error = Virtual machine does not exist, code = 1 (Failed with error noVM and code 1) 2017-10-01 16:01:33,819-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] EVENT_ID: USER_FAILED_STOP_VM(56), Correlation ID: 2bf4b242-4939-4e61-976b-eda93f156af5, Job ID: a6d33241-cee5-40a0-953c-720ded0d8e29, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to power off VM cape (Host: virt4, User: admin@internal). 2017-10-01 16:01:33,842-04 INFO [org.ovirt.engine.core.bll.StopVmCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] Running command: StopVmCommand internal: false. Entities affected : ID: ddf37076-e795-4c87-a4fb-9efcaae76d72 Type: VMAction group STOP_VM with role type USER 2017-10-01 16:01:33,858-04 INFO [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] START, DestroyVmVDSCommand( DestroyVmVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vmId='ddf37076-e795-4c87-a4fb-9efcaae76d72', force='false', secondsToWait='0', gracefully='false', reason='', ignoreNoVm='false'}), log id: 71bc06d8 2017-10-01 16:01:33,858-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] START, DestroyVDSCommand(HostName = virt2, DestroyVmVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vmId='ddf37076-e795-4c87-a4fb-9efcaae76d72', force='false', secondsToWait='0', gracefully='false', reason='', ignoreNoVm='false'}), log id: 2b435fb8 2017-10-01 16:01:33,863-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [4e22b08b] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command HSMGetAllTasksStatusesVDS failed: Not SPM 2017-10-01 16:01:33,863-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler7) [4e22b08b] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand' return value ' 2017-10-01 16:01:33,863-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler7) [4e22b08b] HostName = virt2 2017-10-01 16:01:33,863-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler7) [4e22b08b] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = virt2, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e'})' execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM 2017-10-01 16:01:33,930-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [4e22b08b] START, ConnectStoragePoolVDSCommand(HostName = virt1, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vdsId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 74ca879 2017-10-01 16:01:33,930-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [4e22b08b] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:01:34,035-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-25) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:01:34,035-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:01:34,035-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] HostName = virt2 2017-10-01 16:01:34,035-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] Command 'ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:01:34,035-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] FINISH, ConnectStoragePoolVDSCommand, log id: 29d7b4ea 2017-10-01 16:01:34,035-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-6-thread-25) [] Could not connect host 'virt2' to pool 'EECS-Primary', as the master domain is in inactive/unknown status - not failing the operation 2017-10-01 16:01:34,044-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [4e22b08b] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt1 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:01:34,044-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [4e22b08b] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:01:34,044-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [4e22b08b] HostName = virt1 2017-10-01 16:01:34,045-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [4e22b08b] Command 'ConnectStoragePoolVDSCommand(HostName = virt1, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vdsId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:01:34,045-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [4e22b08b] FINISH, ConnectStoragePoolVDSCommand, log id: 74ca879 2017-10-01 16:01:34,045-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler7) [4e22b08b] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:01:34,186-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [4e22b08b] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt1 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:01:34,186-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler7) [4e22b08b] Command 'SpmStatusVDSCommand(HostName = virt1, SpmStatusVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:01:34,186-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler7) [4e22b08b] Irs placed on server '45f8b331-842e-48e7-9df8-56adddb93836' failed. Proceed Failover 2017-10-01 16:01:34,244-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [4e22b08b] START, ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 4da7e45f 2017-10-01 16:01:34,245-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [4e22b08b] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:01:34,324-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [2dfdaa98] EVENT_ID: FENCE_OPERATION_USING_AGENT_AND_PROXY_STARTED(9,020), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Executing power management status on Host virt2 using Proxy Host virt4 and Fence Agent ipmilan:virt2-idrac. 2017-10-01 16:01:34,325-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler8) [2dfdaa98] START, FenceVdsVDSCommand(HostName = virt4, FenceVdsVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', targetVdsId='a0ccfd11-3402-4eba-a119-4987561b738e', action='STATUS', agent='FenceAgent:{id='7c392b88-1692-461f-9248-12f13b0e0b01', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', order='1', type='ipmilan', ip='virt2-idrac', port='null', user='root', password='***', encryptOptions='false', options='privlvl=OPERATOR, delay=10, lanplus=1'}', policy='null'}), log id: 28d8f5c2 2017-10-01 16:01:34,434-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [4e22b08b] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:01:34,434-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [4e22b08b] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:01:34,434-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [4e22b08b] HostName = virt4 2017-10-01 16:01:34,434-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [4e22b08b] Command 'ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:01:34,434-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [4e22b08b] FINISH, ConnectStoragePoolVDSCommand, log id: 4da7e45f 2017-10-01 16:01:34,434-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler7) [4e22b08b] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:01:34,437-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] FINISH, DestroyVDSCommand, log id: 2b435fb8 2017-10-01 16:01:34,437-04 INFO [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] FINISH, DestroyVmVDSCommand, log id: 71bc06d8 2017-10-01 16:01:34,444-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] EVENT_ID: USER_STOP_VM(33), Correlation ID: 2bf4b242-4939-4e61-976b-eda93f156af5, Job ID: bf508a44-27d5-4946-951a-959ee4a53e5b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM dc1 powered off by admin@internal (Host: virt2). 2017-10-01 16:01:34,457-04 INFO [org.ovirt.engine.core.bll.StopVmCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] Running command: StopVmCommand internal: false. Entities affected : ID: c8682859-9e55-460a-9ef0-d03e4a85f50e Type: VMAction group STOP_VM with role type USER 2017-10-01 16:01:34,494-04 INFO [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] START, DestroyVmVDSCommand( DestroyVmVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vmId='c8682859-9e55-460a-9ef0-d03e4a85f50e', force='false', secondsToWait='0', gracefully='false', reason='', ignoreNoVm='false'}), log id: 6c2fbca0 2017-10-01 16:01:34,494-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] START, DestroyVDSCommand(HostName = virt2, DestroyVmVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vmId='c8682859-9e55-460a-9ef0-d03e4a85f50e', force='false', secondsToWait='0', gracefully='false', reason='', ignoreNoVm='false'}), log id: 6d0c8ab2 2017-10-01 16:01:34,495-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler8) [2dfdaa98] FINISH, FenceVdsVDSCommand, return: FenceOperationResult:{status='SUCCESS', powerStatus='ON', message=''}, log id: 28d8f5c2 2017-10-01 16:01:34,509-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler8) [2dfdaa98] START, SetMOMPolicyParametersVDSCommand(HostName = virt2, MomPolicyVDSParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e'}), log id: 61835c86 2017-10-01 16:01:34,590-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [4e22b08b] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:01:34,590-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler7) [4e22b08b] Command 'SpmStatusVDSCommand(HostName = virt4, SpmStatusVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:01:34,590-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler7) [4e22b08b] Irs placed on server '5ae29ed2-9c0f-4754-8df3-b43d14726846' failed. Proceed Failover 2017-10-01 16:01:34,645-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [4e22b08b] START, ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 149235b8 2017-10-01 16:01:34,645-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [4e22b08b] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:01:34,645-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler8) [2dfdaa98] FINISH, SetMOMPolicyParametersVDSCommand, log id: 61835c86 2017-10-01 16:01:34,665-04 INFO [org.ovirt.engine.core.bll.hostdev.RefreshHostDevicesCommand] (DefaultQuartzScheduler8) [74d3cf78] Running command: RefreshHostDevicesCommand internal: true. Entities affected : ID: a0ccfd11-3402-4eba-a119-4987561b738e Type: VDSAction group MANIPULATE_HOST with role type ADMIN 2017-10-01 16:01:35,099-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] FINISH, DestroyVDSCommand, log id: 6d0c8ab2 2017-10-01 16:01:35,099-04 INFO [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] FINISH, DestroyVmVDSCommand, log id: 6c2fbca0 2017-10-01 16:01:35,103-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [4e22b08b] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:01:35,103-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [4e22b08b] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:01:35,103-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [4e22b08b] HostName = virt3 2017-10-01 16:01:35,103-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [4e22b08b] Command 'ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:01:35,103-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [4e22b08b] FINISH, ConnectStoragePoolVDSCommand, log id: 149235b8 2017-10-01 16:01:35,103-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler7) [4e22b08b] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:01:35,103-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] EVENT_ID: USER_STOP_VM(33), Correlation ID: 2bf4b242-4939-4e61-976b-eda93f156af5, Job ID: 1fa6a507-6ce7-4841-9c23-4f71c596cf08, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM dhcp1 powered off by admin@internal (Host: virt2). 2017-10-01 16:01:35,115-04 INFO [org.ovirt.engine.core.bll.StopVmCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] Running command: StopVmCommand internal: false. Entities affected : ID: 4ba80442-b361-4026-ae8c-895fd28042b0 Type: VMAction group STOP_VM with role type USER 2017-10-01 16:01:35,130-04 INFO [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] START, DestroyVmVDSCommand( DestroyVmVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vmId='4ba80442-b361-4026-ae8c-895fd28042b0', force='false', secondsToWait='0', gracefully='false', reason='', ignoreNoVm='false'}), log id: 3a358946 2017-10-01 16:01:35,131-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] START, DestroyVDSCommand(HostName = virt2, DestroyVmVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vmId='4ba80442-b361-4026-ae8c-895fd28042b0', force='false', secondsToWait='0', gracefully='false', reason='', ignoreNoVm='false'}), log id: 6d07e547 2017-10-01 16:01:35,223-04 INFO [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-19) [fdbf26f] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: c8653dba-63ec-4447-872c-0ffcc6af9c0f Type: Storage 2017-10-01 16:01:35,248-04 WARN [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-19) [fdbf26f] skipping connect and refresh for all hosts, last master 'true', command status 'true' 2017-10-01 16:01:35,260-04 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-6-thread-5) [4e22b08b] Finished reconstruct for pool 'aca61eac-ed8a-484b-bbee-3645148f6405'. Clearing event queue 2017-10-01 16:01:35,509-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmsStatisticsFetcher] (DefaultQuartzScheduler9) [8308c4b] Fetched 9 VMs from VDS 'a0ccfd11-3402-4eba-a119-4987561b738e' 2017-10-01 16:01:35,509-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler9) [8308c4b] VM 'db3e08fb-c111-442d-aeec-1507da30f486'(print) moved from 'NotResponding' --> 'Paused' 2017-10-01 16:01:35,581-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [8308c4b] EVENT_ID: VM_PAUSED(1,025), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM print has been paused. 2017-10-01 16:01:35,602-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [8308c4b] EVENT_ID: VM_PAUSED_ERROR(139), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM print has been paused due to unknown storage error. 2017-10-01 16:01:35,603-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler9) [8308c4b] VM 'ae56c31c-71f7-433a-b198-a1c2c846325e'(forum) moved from 'NotResponding' --> 'Paused' 2017-10-01 16:01:35,624-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [8308c4b] EVENT_ID: VM_PAUSED(1,025), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM forum has been paused. 2017-10-01 16:01:35,645-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [8308c4b] EVENT_ID: VM_PAUSED_ERROR(139), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM forum has been paused due to unknown storage error. 2017-10-01 16:01:35,645-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler9) [8308c4b] VM 'bfe9a524-2238-4bab-a91f-5dc6ed270896'(pcmode) moved from 'NotResponding' --> 'Paused' 2017-10-01 16:01:35,667-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [8308c4b] EVENT_ID: VM_PAUSED(1,025), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM pcmode has been paused. 2017-10-01 16:01:35,687-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [8308c4b] EVENT_ID: VM_PAUSED_ERROR(139), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM pcmode has been paused due to unknown storage error. 2017-10-01 16:01:35,687-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler9) [8308c4b] VM '1325cebf-834e-4571-a3e1-82beb0021ffe'(dns1) moved from 'NotResponding' --> 'Paused' 2017-10-01 16:01:35,727-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] FINISH, DestroyVDSCommand, log id: 6d07e547 2017-10-01 16:01:35,727-04 INFO [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] FINISH, DestroyVmVDSCommand, log id: 3a358946 2017-10-01 16:01:35,731-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] EVENT_ID: USER_STOP_VM(33), Correlation ID: 2bf4b242-4939-4e61-976b-eda93f156af5, Job ID: 21247816-8aa4-4bf0-a9cb-fefd79fa7be4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM dist powered off by admin@internal (Host: virt2). 2017-10-01 16:01:35,743-04 INFO [org.ovirt.engine.core.bll.StopVmCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] Running command: StopVmCommand internal: false. Entities affected : ID: 1325cebf-834e-4571-a3e1-82beb0021ffe Type: VMAction group STOP_VM with role type USER 2017-10-01 16:01:35,753-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [8308c4b] EVENT_ID: VM_PAUSED(1,025), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM dns1 has been paused. 2017-10-01 16:01:35,758-04 INFO [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] START, DestroyVmVDSCommand( DestroyVmVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vmId='1325cebf-834e-4571-a3e1-82beb0021ffe', force='false', secondsToWait='0', gracefully='false', reason='', ignoreNoVm='false'}), log id: 4389cffa 2017-10-01 16:01:35,758-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] START, DestroyVDSCommand(HostName = virt2, DestroyVmVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vmId='1325cebf-834e-4571-a3e1-82beb0021ffe', force='false', secondsToWait='0', gracefully='false', reason='', ignoreNoVm='false'}), log id: 484ab109 2017-10-01 16:01:35,823-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [8308c4b] EVENT_ID: VM_PAUSED_ERROR(139), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM dns1 has been paused due to unknown storage error. 2017-10-01 16:01:35,824-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler9) [8308c4b] VM 'a82b4cc3-6ef6-4a8e-97ef-1d4ea1b7e07c'(indigo) moved from 'NotResponding' --> 'Paused' 2017-10-01 16:01:35,847-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [8308c4b] EVENT_ID: VM_PAUSED(1,025), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM indigo has been paused. 2017-10-01 16:01:35,872-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [8308c4b] EVENT_ID: VM_PAUSED_ERROR(139), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM indigo has been paused due to unknown storage error. 2017-10-01 16:01:35,872-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler9) [8308c4b] VM '3bb83ad5-dc00-445d-8708-2db18c53e0e6'(www-vhost) moved from 'NotResponding' --> 'Paused' 2017-10-01 16:01:35,894-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [8308c4b] EVENT_ID: VM_PAUSED(1,025), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM www-vhost has been paused. 2017-10-01 16:01:35,915-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [8308c4b] EVENT_ID: VM_PAUSED_ERROR(139), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM www-vhost has been paused due to unknown storage error. 2017-10-01 16:01:35,916-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler9) [8308c4b] VM 'a8d4e149-55ec-4961-a414-d7d1e1b05a59'(fix) moved from 'NotResponding' --> 'Paused' 2017-10-01 16:01:35,937-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [8308c4b] EVENT_ID: VM_PAUSED(1,025), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM fix has been paused. 2017-10-01 16:01:35,958-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [8308c4b] EVENT_ID: VM_PAUSED_ERROR(139), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM fix has been paused due to unknown storage error. 2017-10-01 16:01:35,958-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler9) [8308c4b] VM 'aa487207-7ff4-465a-9d9b-2a103d50dc77'(ftp1) moved from 'NotResponding' --> 'Paused' 2017-10-01 16:01:35,979-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [8308c4b] EVENT_ID: VM_PAUSED(1,025), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM ftp1 has been paused. 2017-10-01 16:01:36,000-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [8308c4b] EVENT_ID: VM_PAUSED_ERROR(139), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM ftp1 has been paused due to unknown storage error. 2017-10-01 16:01:36,000-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler9) [8308c4b] VM '4ba80442-b361-4026-ae8c-895fd28042b0'(dist) moved from 'NotResponding' --> 'Paused' 2017-10-01 16:01:36,020-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [8308c4b] EVENT_ID: VM_PAUSED(1,025), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM dist has been paused. 2017-10-01 16:01:36,042-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [8308c4b] EVENT_ID: VM_PAUSED_ERROR(139), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM dist has been paused due to unknown storage error. 2017-10-01 16:01:36,042-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler9) [8308c4b] VM 'ddf37076-e795-4c87-a4fb-9efcaae76d72'(dc1) is running in db and not running on VDS 'a0ccfd11-3402-4eba-a119-4987561b738e'(virt2) 2017-10-01 16:01:36,063-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [8308c4b] EVENT_ID: VM_DOWN(61), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM dc1 is down. 2017-10-01 16:01:36,063-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler9) [8308c4b] VM 'c8682859-9e55-460a-9ef0-d03e4a85f50e'(dhcp1) is running in db and not running on VDS 'a0ccfd11-3402-4eba-a119-4987561b738e'(virt2) 2017-10-01 16:01:36,084-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [8308c4b] EVENT_ID: VM_DOWN(61), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM dhcp1 is down. 2017-10-01 16:01:36,095-04 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler9) [8308c4b] Failed to invoke scheduled method poll: null 2017-10-01 16:01:36,100-04 INFO [org.ovirt.engine.core.bll.ProcessDownVmCommand] (org.ovirt.thread.pool-6-thread-27) [85646ee] Running command: ProcessDownVmCommand internal: true. 2017-10-01 16:01:36,117-04 INFO [org.ovirt.engine.core.bll.ProcessDownVmCommand] (org.ovirt.thread.pool-6-thread-27) [5aa57934] Running command: ProcessDownVmCommand internal: true. 2017-10-01 16:01:36,334-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] FINISH, DestroyVDSCommand, log id: 484ab109 2017-10-01 16:01:36,334-04 INFO [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] FINISH, DestroyVmVDSCommand, log id: 4389cffa 2017-10-01 16:01:36,339-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] EVENT_ID: USER_STOP_VM(33), Correlation ID: 2bf4b242-4939-4e61-976b-eda93f156af5, Job ID: c071d6ad-83c3-44e0-8dbe-90e3b115ea43, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM dns1 powered off by admin@internal (Host: virt2). 2017-10-01 16:01:36,351-04 INFO [org.ovirt.engine.core.bll.StopVmCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] Running command: StopVmCommand internal: false. Entities affected : ID: 62ff4ada-ee98-491e-bfb5-7adda7b513ee Type: VMAction group STOP_VM with role type USER 2017-10-01 16:01:36,360-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] EVENT_ID: VM_ALREADY_IN_REQUESTED_STATUS(254), Correlation ID: 2bf4b242-4939-4e61-976b-eda93f156af5, Job ID: d35be84b-1a08-4e2d-9d7f-179d6b017dcb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM eportfolio is already Down, Stop VM was skipped. User: admin@internal. 2017-10-01 16:01:36,372-04 INFO [org.ovirt.engine.core.bll.StopVmCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] Running command: StopVmCommand internal: false. Entities affected : ID: a8d4e149-55ec-4961-a414-d7d1e1b05a59 Type: VMAction group STOP_VM with role type USER 2017-10-01 16:01:36,387-04 INFO [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] START, DestroyVmVDSCommand( DestroyVmVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vmId='a8d4e149-55ec-4961-a414-d7d1e1b05a59', force='false', secondsToWait='0', gracefully='false', reason='', ignoreNoVm='false'}), log id: 6120ce85 2017-10-01 16:01:36,388-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] START, DestroyVDSCommand(HostName = virt2, DestroyVmVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vmId='a8d4e149-55ec-4961-a414-d7d1e1b05a59', force='false', secondsToWait='0', gracefully='false', reason='', ignoreNoVm='false'}), log id: 25e7bc12 2017-10-01 16:01:36,394-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [74d3cf78] EVENT_ID: VDS_DETECTED(13), Correlation ID: 3b19cfcf, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Status of host virt2 was set to Up. 2017-10-01 16:01:36,404-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [74d3cf78] EVENT_ID: VDS_FENCE_STATUS(496), Correlation ID: 3b19cfcf, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host virt2 power management was verified successfully. 2017-10-01 16:01:36,439-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler8) [7cca164b] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: a0ccfd11-3402-4eba-a119-4987561b738e Type: VDS 2017-10-01 16:01:36,983-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] FINISH, DestroyVDSCommand, log id: 25e7bc12 2017-10-01 16:01:36,983-04 INFO [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] FINISH, DestroyVmVDSCommand, log id: 6120ce85 2017-10-01 16:01:36,987-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] EVENT_ID: USER_STOP_VM(33), Correlation ID: 2bf4b242-4939-4e61-976b-eda93f156af5, Job ID: 4eaa4fe3-b1ef-436c-936a-48cae404785b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM fix powered off by admin@internal (Host: virt2). 2017-10-01 16:01:37,000-04 INFO [org.ovirt.engine.core.bll.StopVmCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] Running command: StopVmCommand internal: false. Entities affected : ID: ae56c31c-71f7-433a-b198-a1c2c846325e Type: VMAction group STOP_VM with role type USER 2017-10-01 16:01:37,017-04 INFO [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] START, DestroyVmVDSCommand( DestroyVmVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vmId='ae56c31c-71f7-433a-b198-a1c2c846325e', force='false', secondsToWait='0', gracefully='false', reason='', ignoreNoVm='false'}), log id: 21158bf4 2017-10-01 16:01:37,018-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] START, DestroyVDSCommand(HostName = virt2, DestroyVmVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vmId='ae56c31c-71f7-433a-b198-a1c2c846325e', force='false', secondsToWait='0', gracefully='false', reason='', ignoreNoVm='false'}), log id: 106ee988 2017-10-01 16:01:38,618-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] FINISH, DestroyVDSCommand, log id: 106ee988 2017-10-01 16:01:38,618-04 INFO [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] FINISH, DestroyVmVDSCommand, log id: 21158bf4 2017-10-01 16:01:38,623-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] EVENT_ID: USER_STOP_VM(33), Correlation ID: 2bf4b242-4939-4e61-976b-eda93f156af5, Job ID: 6dc6259b-fe49-47eb-962d-cf2f7b349c73, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM forum powered off by admin@internal (Host: virt2). 2017-10-01 16:01:38,635-04 INFO [org.ovirt.engine.core.bll.StopVmCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] Running command: StopVmCommand internal: false. Entities affected : ID: aa487207-7ff4-465a-9d9b-2a103d50dc77 Type: VMAction group STOP_VM with role type USER 2017-10-01 16:01:38,650-04 INFO [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] START, DestroyVmVDSCommand( DestroyVmVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vmId='aa487207-7ff4-465a-9d9b-2a103d50dc77', force='false', secondsToWait='0', gracefully='false', reason='', ignoreNoVm='false'}), log id: 7f48baae 2017-10-01 16:01:38,650-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] START, DestroyVDSCommand(HostName = virt2, DestroyVmVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vmId='aa487207-7ff4-465a-9d9b-2a103d50dc77', force='false', secondsToWait='0', gracefully='false', reason='', ignoreNoVm='false'}), log id: 61573d76 2017-10-01 16:01:40,200-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] FINISH, DestroyVDSCommand, log id: 61573d76 2017-10-01 16:01:40,200-04 INFO [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] FINISH, DestroyVmVDSCommand, log id: 7f48baae 2017-10-01 16:01:40,205-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] EVENT_ID: USER_STOP_VM(33), Correlation ID: 2bf4b242-4939-4e61-976b-eda93f156af5, Job ID: 8018d504-a904-47c0-bb29-60dc29ff807c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM ftp1 powered off by admin@internal (Host: virt2). 2017-10-01 16:01:40,216-04 INFO [org.ovirt.engine.core.bll.StopVmCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] Running command: StopVmCommand internal: false. Entities affected : ID: 911d6883-607c-464e-b46b-8e6141bdd2de Type: VMAction group STOP_VM with role type USER 2017-10-01 16:01:40,232-04 INFO [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] START, DestroyVmVDSCommand( DestroyVmVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vmId='911d6883-607c-464e-b46b-8e6141bdd2de', force='false', secondsToWait='0', gracefully='false', reason='', ignoreNoVm='false'}), log id: 626abd03 2017-10-01 16:01:40,232-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] START, DestroyVDSCommand(HostName = virt4, DestroyVmVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vmId='911d6883-607c-464e-b46b-8e6141bdd2de', force='false', secondsToWait='0', gracefully='false', reason='', ignoreNoVm='false'}), log id: 2ae5ecc9 2017-10-01 16:01:40,787-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] FINISH, DestroyVDSCommand, log id: 2ae5ecc9 2017-10-01 16:01:40,787-04 INFO [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] FINISH, DestroyVmVDSCommand, log id: 626abd03 2017-10-01 16:01:40,792-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] EVENT_ID: USER_STOP_VM(33), Correlation ID: 2bf4b242-4939-4e61-976b-eda93f156af5, Job ID: 1144807c-4d0b-49a5-bf3e-a4d428bb346c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM hdb powered off by admin@internal (Host: virt4). 2017-10-01 16:01:40,803-04 INFO [org.ovirt.engine.core.bll.StopVmCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] Running command: StopVmCommand internal: false. Entities affected : ID: a82b4cc3-6ef6-4a8e-97ef-1d4ea1b7e07c Type: VMAction group STOP_VM with role type USER 2017-10-01 16:01:40,819-04 INFO [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] START, DestroyVmVDSCommand( DestroyVmVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vmId='a82b4cc3-6ef6-4a8e-97ef-1d4ea1b7e07c', force='false', secondsToWait='0', gracefully='false', reason='', ignoreNoVm='false'}), log id: 1b97290d 2017-10-01 16:01:40,819-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] START, DestroyVDSCommand(HostName = virt2, DestroyVmVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vmId='a82b4cc3-6ef6-4a8e-97ef-1d4ea1b7e07c', force='false', secondsToWait='0', gracefully='false', reason='', ignoreNoVm='false'}), log id: 5130d118 2017-10-01 16:01:41,370-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] FINISH, DestroyVDSCommand, log id: 5130d118 2017-10-01 16:01:41,370-04 INFO [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] FINISH, DestroyVmVDSCommand, log id: 1b97290d 2017-10-01 16:01:41,375-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] EVENT_ID: USER_STOP_VM(33), Correlation ID: 2bf4b242-4939-4e61-976b-eda93f156af5, Job ID: cc6ce4f4-081d-427f-b3fe-3dc561f4cb26, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM indigo powered off by admin@internal (Host: virt2). 2017-10-01 16:01:41,388-04 INFO [org.ovirt.engine.core.bll.StopVmCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] Running command: StopVmCommand internal: false. Entities affected : ID: 922e2017-0fd6-4ac8-a204-449bfcd3f2c5 Type: VMAction group STOP_VM with role type USER 2017-10-01 16:01:41,402-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] EVENT_ID: VM_ALREADY_IN_REQUESTED_STATUS(254), Correlation ID: 2bf4b242-4939-4e61-976b-eda93f156af5, Job ID: 485a8200-eeb5-42ab-842d-05609072e773, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM ingtest is already Down, Stop VM was skipped. User: admin@internal. 2017-10-01 16:01:41,424-04 INFO [org.ovirt.engine.core.bll.StopVmCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] Running command: StopVmCommand internal: false. Entities affected : ID: 899e3a1c-dcc4-4426-9dd4-0f1e8b94a2b8 Type: VMAction group STOP_VM with role type USER 2017-10-01 16:01:41,440-04 INFO [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] START, DestroyVmVDSCommand( DestroyVmVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vmId='899e3a1c-dcc4-4426-9dd4-0f1e8b94a2b8', force='false', secondsToWait='0', gracefully='false', reason='', ignoreNoVm='false'}), log id: 30e04795 2017-10-01 16:01:41,440-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] START, DestroyVDSCommand(HostName = virt1, DestroyVmVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vmId='899e3a1c-dcc4-4426-9dd4-0f1e8b94a2b8', force='false', secondsToWait='0', gracefully='false', reason='', ignoreNoVm='false'}), log id: 7e75c36a 2017-10-01 16:01:41,994-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-23) [6d81b0a3] FINISH, ConnectStorageServerVDSCommand, return: {13cf1e67-9961-4a25-a41f-cc76b4ab793d=0}, log id: 13d9ef27 2017-10-01 16:01:41,997-04 INFO [org.ovirt.engine.core.bll.storage.connection.ConnectDomainToStorageCommand] (DefaultQuartzScheduler6) [535c1f37] ConnectDomainToStorage. After Connect all hosts to pool. Time: Sun Oct 01 16:01:41 EDT 2017 2017-10-01 16:01:41,999-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [535c1f37] Autorecovering storage domains id: f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2 2017-10-01 16:01:42,002-04 INFO [org.ovirt.engine.core.bll.storage.connection.ConnectDomainToStorageCommand] (DefaultQuartzScheduler6) [37e7838a] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2 Type: Storage 2017-10-01 16:01:42,002-04 INFO [org.ovirt.engine.core.bll.storage.connection.ConnectDomainToStorageCommand] (DefaultQuartzScheduler6) [37e7838a] ConnectDomainToStorage. Before Connect all hosts to pool. Time: Sun Oct 01 16:01:42 EDT 2017 2017-10-01 16:01:42,076-04 INFO [org.ovirt.engine.core.bll.storage.connection.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-6-thread-42) [179f3d4c] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN 2017-10-01 16:01:42,076-04 INFO [org.ovirt.engine.core.bll.storage.connection.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-6-thread-39) [3c2c9939] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN 2017-10-01 16:01:42,077-04 INFO [org.ovirt.engine.core.bll.storage.connection.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-6-thread-31) [73e5220a] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN 2017-10-01 16:01:42,077-04 INFO [org.ovirt.engine.core.bll.storage.connection.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-6-thread-22) [151057d1] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN 2017-10-01 16:01:42,092-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-22) [151057d1] START, ConnectStorageServerVDSCommand(HostName = virt3, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='00000000-0000-0000-0000-000000000000', storageType='NFS', connectionList='[StorageServerConnections:{id='417fdd51-dbcd-41d9-9a05-ef4d9f485d13', connection='virtfs-fs:/nfs/export', iqn='null', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: cd9a136 2017-10-01 16:01:42,094-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] FINISH, DestroyVDSCommand, log id: 7e75c36a 2017-10-01 16:01:42,094-04 INFO [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] FINISH, DestroyVmVDSCommand, log id: 30e04795 2017-10-01 16:01:42,100-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] EVENT_ID: USER_STOP_VM(33), Correlation ID: 2bf4b242-4939-4e61-976b-eda93f156af5, Job ID: 01263e1d-4b66-448f-8347-19def017f139, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM ipa powered off by admin@internal (Host: virt1). 2017-10-01 16:01:42,100-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-42) [179f3d4c] START, ConnectStorageServerVDSCommand(HostName = virt4, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='00000000-0000-0000-0000-000000000000', storageType='NFS', connectionList='[StorageServerConnections:{id='417fdd51-dbcd-41d9-9a05-ef4d9f485d13', connection='virtfs-fs:/nfs/export', iqn='null', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 63a7986 2017-10-01 16:01:42,110-04 INFO [org.ovirt.engine.core.bll.StopVmCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] Running command: StopVmCommand internal: false. Entities affected : ID: 77766cb4-0625-4432-a16f-def5e702102a Type: VMAction group STOP_VM with role type USER 2017-10-01 16:01:42,113-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-31) [73e5220a] START, ConnectStorageServerVDSCommand(HostName = virt2, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='00000000-0000-0000-0000-000000000000', storageType='NFS', connectionList='[StorageServerConnections:{id='417fdd51-dbcd-41d9-9a05-ef4d9f485d13', connection='virtfs-fs:/nfs/export', iqn='null', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 46f5a2be 2017-10-01 16:01:42,114-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-22) [151057d1] FINISH, ConnectStorageServerVDSCommand, return: {417fdd51-dbcd-41d9-9a05-ef4d9f485d13=0}, log id: cd9a136 2017-10-01 16:01:42,120-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-39) [3c2c9939] START, ConnectStorageServerVDSCommand(HostName = virt1, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='00000000-0000-0000-0000-000000000000', storageType='NFS', connectionList='[StorageServerConnections:{id='417fdd51-dbcd-41d9-9a05-ef4d9f485d13', connection='virtfs-fs:/nfs/export', iqn='null', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 7dfb7239 2017-10-01 16:01:42,121-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-31) [73e5220a] FINISH, ConnectStorageServerVDSCommand, return: {417fdd51-dbcd-41d9-9a05-ef4d9f485d13=0}, log id: 46f5a2be 2017-10-01 16:01:42,149-04 INFO [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] START, DestroyVmVDSCommand( DestroyVmVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vmId='77766cb4-0625-4432-a16f-def5e702102a', force='false', secondsToWait='0', gracefully='false', reason='', ignoreNoVm='false'}), log id: 25a7b545 2017-10-01 16:01:42,149-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] START, DestroyVDSCommand(HostName = virt1, DestroyVmVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vmId='77766cb4-0625-4432-a16f-def5e702102a', force='false', secondsToWait='0', gracefully='false', reason='', ignoreNoVm='false'}), log id: 25c11875 2017-10-01 16:01:42,149-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-42) [179f3d4c] FINISH, ConnectStorageServerVDSCommand, return: {417fdd51-dbcd-41d9-9a05-ef4d9f485d13=0}, log id: 63a7986 2017-10-01 16:01:42,714-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-39) [3c2c9939] FINISH, ConnectStorageServerVDSCommand, return: {417fdd51-dbcd-41d9-9a05-ef4d9f485d13=0}, log id: 7dfb7239 2017-10-01 16:01:42,717-04 INFO [org.ovirt.engine.core.bll.storage.connection.ConnectDomainToStorageCommand] (DefaultQuartzScheduler6) [37e7838a] ConnectDomainToStorage. After Connect all hosts to pool. Time: Sun Oct 01 16:01:42 EDT 2017 2017-10-01 16:01:42,718-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [37e7838a] Autorecovering storage domains id: c8653dba-63ec-4447-872c-0ffcc6af9c0f 2017-10-01 16:01:42,722-04 INFO [org.ovirt.engine.core.bll.storage.connection.ConnectDomainToStorageCommand] (DefaultQuartzScheduler6) [3eed4b24] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: c8653dba-63ec-4447-872c-0ffcc6af9c0f Type: Storage 2017-10-01 16:01:42,722-04 INFO [org.ovirt.engine.core.bll.storage.connection.ConnectDomainToStorageCommand] (DefaultQuartzScheduler6) [3eed4b24] ConnectDomainToStorage. Before Connect all hosts to pool. Time: Sun Oct 01 16:01:42 EDT 2017 2017-10-01 16:01:42,789-04 INFO [org.ovirt.engine.core.bll.storage.connection.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-6-thread-17) [11288cd1] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN 2017-10-01 16:01:42,794-04 INFO [org.ovirt.engine.core.bll.storage.connection.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-6-thread-30) [652e5098] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN 2017-10-01 16:01:42,794-04 INFO [org.ovirt.engine.core.bll.storage.connection.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-6-thread-43) [65cb9e13] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN 2017-10-01 16:01:42,794-04 INFO [org.ovirt.engine.core.bll.storage.connection.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-6-thread-12) [1957ec02] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN 2017-10-01 16:01:42,805-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-17) [11288cd1] START, ConnectStorageServerVDSCommand(HostName = virt1, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='00000000-0000-0000-0000-000000000000', storageType='NFS', connectionList='[StorageServerConnections:{id='5e557659-6817-40fa-9a55-7458125404a6', connection='virtfs-fs:/nfs/data', iqn='null', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 491932d4 2017-10-01 16:01:42,807-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] FINISH, DestroyVDSCommand, log id: 25c11875 2017-10-01 16:01:42,807-04 INFO [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] FINISH, DestroyVmVDSCommand, log id: 25a7b545 2017-10-01 16:01:42,809-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-30) [652e5098] START, ConnectStorageServerVDSCommand(HostName = virt2, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='00000000-0000-0000-0000-000000000000', storageType='NFS', connectionList='[StorageServerConnections:{id='5e557659-6817-40fa-9a55-7458125404a6', connection='virtfs-fs:/nfs/data', iqn='null', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 4f3bf309 2017-10-01 16:01:42,809-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-43) [65cb9e13] START, ConnectStorageServerVDSCommand(HostName = virt3, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='00000000-0000-0000-0000-000000000000', storageType='NFS', connectionList='[StorageServerConnections:{id='5e557659-6817-40fa-9a55-7458125404a6', connection='virtfs-fs:/nfs/data', iqn='null', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 5803c70b 2017-10-01 16:01:42,810-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-12) [1957ec02] START, ConnectStorageServerVDSCommand(HostName = virt4, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='00000000-0000-0000-0000-000000000000', storageType='NFS', connectionList='[StorageServerConnections:{id='5e557659-6817-40fa-9a55-7458125404a6', connection='virtfs-fs:/nfs/data', iqn='null', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 495b7f8f 2017-10-01 16:01:42,811-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-17) [11288cd1] FINISH, ConnectStorageServerVDSCommand, return: {5e557659-6817-40fa-9a55-7458125404a6=0}, log id: 491932d4 2017-10-01 16:01:42,811-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] EVENT_ID: USER_STOP_VM(33), Correlation ID: 2bf4b242-4939-4e61-976b-eda93f156af5, Job ID: 7575c4fe-61f2-4d4b-953c-f9b4efaf1cb2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM labtest1 powered off by admin@internal (Host: virt1). 2017-10-01 16:01:42,814-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-12) [1957ec02] FINISH, ConnectStorageServerVDSCommand, return: {5e557659-6817-40fa-9a55-7458125404a6=0}, log id: 495b7f8f 2017-10-01 16:01:42,815-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-43) [65cb9e13] FINISH, ConnectStorageServerVDSCommand, return: {5e557659-6817-40fa-9a55-7458125404a6=0}, log id: 5803c70b 2017-10-01 16:01:42,823-04 INFO [org.ovirt.engine.core.bll.StopVmCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] Running command: StopVmCommand internal: false. Entities affected : ID: cc1dbe93-bb80-451e-8b3b-c9cc5e90432f Type: VMAction group STOP_VM with role type USER 2017-10-01 16:01:42,839-04 INFO [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] START, DestroyVmVDSCommand( DestroyVmVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vmId='cc1dbe93-bb80-451e-8b3b-c9cc5e90432f', force='false', secondsToWait='0', gracefully='false', reason='', ignoreNoVm='false'}), log id: 66d0b551 2017-10-01 16:01:42,839-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] START, DestroyVDSCommand(HostName = virt3, DestroyVmVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vmId='cc1dbe93-bb80-451e-8b3b-c9cc5e90432f', force='false', secondsToWait='0', gracefully='false', reason='', ignoreNoVm='false'}), log id: ec2184c 2017-10-01 16:01:42,839-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-30) [652e5098] FINISH, ConnectStorageServerVDSCommand, return: {5e557659-6817-40fa-9a55-7458125404a6=0}, log id: 4f3bf309 2017-10-01 16:01:42,842-04 INFO [org.ovirt.engine.core.bll.storage.connection.ConnectDomainToStorageCommand] (DefaultQuartzScheduler6) [3eed4b24] ConnectDomainToStorage. After Connect all hosts to pool. Time: Sun Oct 01 16:01:42 EDT 2017 2017-10-01 16:01:43,569-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] FINISH, DestroyVDSCommand, log id: ec2184c 2017-10-01 16:01:43,569-04 INFO [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] FINISH, DestroyVmVDSCommand, log id: 66d0b551 2017-10-01 16:01:43,579-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] EVENT_ID: USER_STOP_VM(33), Correlation ID: 2bf4b242-4939-4e61-976b-eda93f156af5, Job ID: 0e380691-c994-44db-869f-7ce63f95af32, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM license powered off by admin@internal (Host: virt3). 2017-10-01 16:01:43,593-04 INFO [org.ovirt.engine.core.bll.StopVmCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] Running command: StopVmCommand internal: false. Entities affected : ID: bfe9a524-2238-4bab-a91f-5dc6ed270896 Type: VMAction group STOP_VM with role type USER 2017-10-01 16:01:43,609-04 INFO [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] START, DestroyVmVDSCommand( DestroyVmVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vmId='bfe9a524-2238-4bab-a91f-5dc6ed270896', force='false', secondsToWait='0', gracefully='false', reason='', ignoreNoVm='false'}), log id: 17b71519 2017-10-01 16:01:43,610-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] START, DestroyVDSCommand(HostName = virt2, DestroyVmVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vmId='bfe9a524-2238-4bab-a91f-5dc6ed270896', force='false', secondsToWait='0', gracefully='false', reason='', ignoreNoVm='false'}), log id: 27ff193 2017-10-01 16:01:44,246-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] FINISH, DestroyVDSCommand, log id: 27ff193 2017-10-01 16:01:44,246-04 INFO [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] FINISH, DestroyVmVDSCommand, log id: 17b71519 2017-10-01 16:01:44,253-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] EVENT_ID: USER_STOP_VM(33), Correlation ID: 2bf4b242-4939-4e61-976b-eda93f156af5, Job ID: 77189e40-f03d-42f3-88f2-ebf095f4d1fb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM pcmode powered off by admin@internal (Host: virt2). 2017-10-01 16:01:44,266-04 INFO [org.ovirt.engine.core.bll.StopVmCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] Running command: StopVmCommand internal: false. Entities affected : ID: db3e08fb-c111-442d-aeec-1507da30f486 Type: VMAction group STOP_VM with role type USER 2017-10-01 16:01:44,281-04 INFO [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] START, DestroyVmVDSCommand( DestroyVmVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vmId='db3e08fb-c111-442d-aeec-1507da30f486', force='false', secondsToWait='0', gracefully='false', reason='', ignoreNoVm='false'}), log id: 45b69f9b 2017-10-01 16:01:44,281-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] START, DestroyVDSCommand(HostName = virt2, DestroyVmVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vmId='db3e08fb-c111-442d-aeec-1507da30f486', force='false', secondsToWait='0', gracefully='false', reason='', ignoreNoVm='false'}), log id: 5d7b8cbe 2017-10-01 16:01:45,213-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] FINISH, DestroyVDSCommand, log id: 5d7b8cbe 2017-10-01 16:01:45,213-04 INFO [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] FINISH, DestroyVmVDSCommand, log id: 45b69f9b 2017-10-01 16:01:45,219-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] EVENT_ID: USER_STOP_VM(33), Correlation ID: 2bf4b242-4939-4e61-976b-eda93f156af5, Job ID: 390a98f3-d1ed-4827-a7eb-a53f1f5fc9aa, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM print powered off by admin@internal (Host: virt2). 2017-10-01 16:01:45,229-04 INFO [org.ovirt.engine.core.bll.StopVmCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] Running command: StopVmCommand internal: false. Entities affected : ID: 5885c43a-b133-45dc-b8d5-1aa2a4a19579 Type: VMAction group STOP_VM with role type USER 2017-10-01 16:01:45,245-04 INFO [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] START, DestroyVmVDSCommand( DestroyVmVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vmId='5885c43a-b133-45dc-b8d5-1aa2a4a19579', force='false', secondsToWait='0', gracefully='false', reason='', ignoreNoVm='false'}), log id: 105b33e3 2017-10-01 16:01:45,245-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] START, DestroyVDSCommand(HostName = virt4, DestroyVmVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vmId='5885c43a-b133-45dc-b8d5-1aa2a4a19579', force='false', secondsToWait='0', gracefully='false', reason='', ignoreNoVm='false'}), log id: 15f8a2d0 2017-10-01 16:01:45,634-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] FINISH, DestroyVDSCommand, log id: 15f8a2d0 2017-10-01 16:01:45,635-04 INFO [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] FINISH, DestroyVmVDSCommand, log id: 105b33e3 2017-10-01 16:01:45,639-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [7194fe99] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command HSMGetAllTasksStatusesVDS failed: Not SPM: () 2017-10-01 16:01:45,639-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler8) [7194fe99] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand' return value ' 2017-10-01 16:01:45,640-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler8) [7194fe99] HostName = virt3 2017-10-01 16:01:45,640-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler8) [7194fe99] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = virt3, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736'})' execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () 2017-10-01 16:01:45,641-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] EVENT_ID: USER_STOP_VM(33), Correlation ID: 2bf4b242-4939-4e61-976b-eda93f156af5, Job ID: 4e3e9055-9e87-44e9-b02f-52da55e15299, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM prs powered off by admin@internal (Host: virt4). 2017-10-01 16:01:45,653-04 INFO [org.ovirt.engine.core.bll.StopVmCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] Running command: StopVmCommand internal: false. Entities affected : ID: c64d80f9-a3bd-44f3-89c4-f1b9b509cdde Type: VMAction group STOP_VM with role type USER 2017-10-01 16:01:45,674-04 INFO [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] START, DestroyVmVDSCommand( DestroyVmVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vmId='c64d80f9-a3bd-44f3-89c4-f1b9b509cdde', force='false', secondsToWait='0', gracefully='false', reason='', ignoreNoVm='false'}), log id: 75ae3179 2017-10-01 16:01:45,674-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] START, DestroyVDSCommand(HostName = virt3, DestroyVmVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vmId='c64d80f9-a3bd-44f3-89c4-f1b9b509cdde', force='false', secondsToWait='0', gracefully='false', reason='', ignoreNoVm='false'}), log id: 43274619 2017-10-01 16:01:45,733-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [7194fe99] START, ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 5cec19f2 2017-10-01 16:01:45,733-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [7194fe99] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:01:46,255-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [7194fe99] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:01:46,255-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [7194fe99] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:01:46,255-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [7194fe99] HostName = virt4 2017-10-01 16:01:46,255-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [7194fe99] Command 'ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:01:46,255-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [7194fe99] FINISH, ConnectStoragePoolVDSCommand, log id: 5cec19f2 2017-10-01 16:01:46,255-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler8) [7194fe99] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:01:46,376-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] FINISH, DestroyVDSCommand, log id: 43274619 2017-10-01 16:01:46,376-04 INFO [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] FINISH, DestroyVmVDSCommand, log id: 75ae3179 2017-10-01 16:01:46,382-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] EVENT_ID: USER_STOP_VM(33), Correlation ID: 2bf4b242-4939-4e61-976b-eda93f156af5, Job ID: 9ce51867-99c6-4d19-9d41-11a8e7bcbda2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM red powered off by admin@internal (Host: virt3). 2017-10-01 16:01:46,395-04 INFO [org.ovirt.engine.core.bll.StopVmCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] Running command: StopVmCommand internal: false. Entities affected : ID: 9ea2821e-557f-45cb-9872-1e6c5aeaf246 Type: VMAction group STOP_VM with role type USER 2017-10-01 16:01:46,411-04 INFO [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] START, DestroyVmVDSCommand( DestroyVmVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vmId='9ea2821e-557f-45cb-9872-1e6c5aeaf246', force='false', secondsToWait='0', gracefully='false', reason='', ignoreNoVm='false'}), log id: 10d9988b 2017-10-01 16:01:46,411-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] START, DestroyVDSCommand(HostName = virt4, DestroyVmVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vmId='9ea2821e-557f-45cb-9872-1e6c5aeaf246', force='false', secondsToWait='0', gracefully='false', reason='', ignoreNoVm='false'}), log id: 3f9d67a3 2017-10-01 16:01:46,415-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [7194fe99] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:01:46,415-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler8) [7194fe99] Command 'SpmStatusVDSCommand(HostName = virt4, SpmStatusVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:01:46,415-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler8) [7194fe99] Irs placed on server '5ae29ed2-9c0f-4754-8df3-b43d14726846' failed. Proceed Failover 2017-10-01 16:01:46,485-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [7194fe99] START, ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 5da247e 2017-10-01 16:01:46,485-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [7194fe99] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:01:46,615-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [7194fe99] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:01:46,615-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [7194fe99] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:01:46,615-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [7194fe99] HostName = virt3 2017-10-01 16:01:46,615-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [7194fe99] Command 'ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:01:46,615-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [7194fe99] FINISH, ConnectStoragePoolVDSCommand, log id: 5da247e 2017-10-01 16:01:46,615-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler8) [7194fe99] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:01:46,724-04 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler4) [753d4fb] Storage Domain 'f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2:export' was reported by Host 'virt1' as Active in Pool 'aca61eac-ed8a-484b-bbee-3645148f6405', moving to active status 2017-10-01 16:01:46,738-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmsStatisticsFetcher] (DefaultQuartzScheduler10) [4e36e836] Fetched 10 VMs from VDS '5ae29ed2-9c0f-4754-8df3-b43d14726846' 2017-10-01 16:01:46,738-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler10) [4e36e836] VM '07331c78-6e09-41b4-9b29-edda32f46351'(samba) moved from 'NotResponding' --> 'Paused' 2017-10-01 16:01:46,760-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [4e36e836] EVENT_ID: VM_PAUSED(1,025), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM samba has been paused. 2017-10-01 16:01:46,781-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [4e36e836] EVENT_ID: VM_PAUSED_ERROR(139), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM samba has been paused due to unknown storage error. 2017-10-01 16:01:46,781-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler10) [4e36e836] VM 'dc66c051-24b7-4cb8-99eb-db398d396ef1'(samba1) moved from 'NotResponding' --> 'Paused' 2017-10-01 16:01:46,782-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler8) [7194fe99] Irs placed on server 'e1e71e9d-a0cd-47db-96c6-f724f6cea736' failed. Proceed Failover 2017-10-01 16:01:46,851-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [4e36e836] EVENT_ID: VM_PAUSED(1,025), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM samba1 has been paused. 2017-10-01 16:01:46,852-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [7194fe99] START, ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 3ef5872f 2017-10-01 16:01:46,852-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [7194fe99] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:01:46,866-04 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-21) [] domain 'f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2:export' in problem 'NOT_REPORTED'. vds: 'virt3' 2017-10-01 16:01:46,879-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [4e36e836] EVENT_ID: VM_PAUSED_ERROR(139), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM samba1 has been paused due to unknown storage error. 2017-10-01 16:01:46,879-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler10) [4e36e836] VM '3e5a5728-8940-4fe7-b4af-11fb8e12f116'(www) moved from 'NotResponding' --> 'Paused' 2017-10-01 16:01:46,900-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [4e36e836] EVENT_ID: VM_PAUSED(1,025), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM www has been paused. 2017-10-01 16:01:46,921-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [4e36e836] EVENT_ID: VM_PAUSED_ERROR(139), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM www has been paused due to unknown storage error. 2017-10-01 16:01:46,921-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler10) [4e36e836] VM 'f87ec08d-0fdf-42ba-8aac-c66c404d7aae'(webapp) moved from 'NotResponding' --> 'Paused' 2017-10-01 16:01:46,942-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [4e36e836] EVENT_ID: VM_PAUSED(1,025), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM webapp has been paused. 2017-10-01 16:01:46,972-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [4e36e836] EVENT_ID: VM_PAUSED_ERROR(139), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM webapp has been paused due to unknown storage error. 2017-10-01 16:01:46,972-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler10) [4e36e836] VM '8391f9a7-7fe7-4025-8fba-6273ae587a52'(archive) moved from 'NotResponding' --> 'Paused' 2017-10-01 16:01:46,972-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [7194fe99] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:01:46,972-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [7194fe99] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:01:46,973-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [7194fe99] HostName = virt2 2017-10-01 16:01:46,973-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [7194fe99] Command 'ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:01:46,973-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [7194fe99] FINISH, ConnectStoragePoolVDSCommand, log id: 3ef5872f 2017-10-01 16:01:46,973-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler8) [7194fe99] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:01:46,995-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [4e36e836] EVENT_ID: VM_PAUSED(1,025), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM archive has been paused. 2017-10-01 16:01:46,999-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] FINISH, DestroyVDSCommand, log id: 3f9d67a3 2017-10-01 16:01:46,999-04 INFO [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] FINISH, DestroyVmVDSCommand, log id: 10d9988b 2017-10-01 16:01:47,014-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] EVENT_ID: USER_STOP_VM(33), Correlation ID: 2bf4b242-4939-4e61-976b-eda93f156af5, Job ID: 7d90d8f8-0b83-4ecd-a5ca-4f1b1f8cfcae, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM rs powered off by admin@internal (Host: virt4). 2017-10-01 16:01:47,018-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [4e36e836] EVENT_ID: VM_PAUSED_ERROR(139), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM archive has been paused due to unknown storage error. 2017-10-01 16:01:47,018-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler10) [4e36e836] VM '1c7e966f-29e8-4788-b64c-86e52f415061'(udb) moved from 'NotResponding' --> 'Paused' 2017-10-01 16:01:47,025-04 INFO [org.ovirt.engine.core.bll.StopVmCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] Running command: StopVmCommand internal: false. Entities affected : ID: 07331c78-6e09-41b4-9b29-edda32f46351 Type: VMAction group STOP_VM with role type USER 2017-10-01 16:01:47,041-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [4e36e836] EVENT_ID: VM_PAUSED(1,025), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM udb has been paused. 2017-10-01 16:01:47,043-04 INFO [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] START, DestroyVmVDSCommand( DestroyVmVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vmId='07331c78-6e09-41b4-9b29-edda32f46351', force='false', secondsToWait='0', gracefully='false', reason='', ignoreNoVm='false'}), log id: 2d373104 2017-10-01 16:01:47,043-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] START, DestroyVDSCommand(HostName = virt4, DestroyVmVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vmId='07331c78-6e09-41b4-9b29-edda32f46351', force='false', secondsToWait='0', gracefully='false', reason='', ignoreNoVm='false'}), log id: 7acf22e2 2017-10-01 16:01:47,062-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [4e36e836] EVENT_ID: VM_PAUSED_ERROR(139), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM udb has been paused due to unknown storage error. 2017-10-01 16:01:47,062-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler10) [4e36e836] VM '9ea2821e-557f-45cb-9872-1e6c5aeaf246'(rs) moved from 'NotResponding' --> 'Paused' 2017-10-01 16:01:47,084-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [4e36e836] EVENT_ID: VM_PAUSED(1,025), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM rs has been paused. 2017-10-01 16:01:47,105-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [4e36e836] EVENT_ID: VM_PAUSED_ERROR(139), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM rs has been paused due to unknown storage error. 2017-10-01 16:01:47,105-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler10) [4e36e836] VM 'd66235b4-7c8d-4da2-a572-82ecc9c53a59'(www-las) moved from 'NotResponding' --> 'Paused' 2017-10-01 16:01:47,125-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [4e36e836] EVENT_ID: VM_PAUSED(1,025), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM www-las has been paused. 2017-10-01 16:01:47,145-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [4e36e836] EVENT_ID: VM_PAUSED_ERROR(139), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM www-las has been paused due to unknown storage error. 2017-10-01 16:01:47,145-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler10) [4e36e836] VM 'ae65a104-ed3f-40e0-ba92-5708cf338918'(vpn2) moved from 'NotResponding' --> 'Paused' 2017-10-01 16:01:47,165-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [4e36e836] EVENT_ID: VM_PAUSED(1,025), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM vpn2 has been paused. 2017-10-01 16:01:47,184-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [4e36e836] EVENT_ID: VM_PAUSED_ERROR(139), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM vpn2 has been paused due to unknown storage error. 2017-10-01 16:01:47,185-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler10) [4e36e836] VM 'd56363ad-9cb1-4a70-a33a-ce439b044d7d'(svn) moved from 'NotResponding' --> 'Paused' 2017-10-01 16:01:47,204-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [4e36e836] EVENT_ID: VM_PAUSED(1,025), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM svn has been paused. 2017-10-01 16:01:47,224-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [4e36e836] EVENT_ID: VM_PAUSED_ERROR(139), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM svn has been paused due to unknown storage error. 2017-10-01 16:01:47,224-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler10) [4e36e836] VM 'bcd5776e-978b-4723-933d-70a80322a796'(cape) is running in db and not running on VDS '5ae29ed2-9c0f-4754-8df3-b43d14726846'(virt4) 2017-10-01 16:01:47,225-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler10) [4e36e836] VM '911d6883-607c-464e-b46b-8e6141bdd2de'(hdb) is running in db and not running on VDS '5ae29ed2-9c0f-4754-8df3-b43d14726846'(virt4) 2017-10-01 16:01:47,245-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [4e36e836] EVENT_ID: VM_DOWN(61), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM hdb is down. 2017-10-01 16:01:47,245-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler10) [4e36e836] VM '5885c43a-b133-45dc-b8d5-1aa2a4a19579'(prs) is running in db and not running on VDS '5ae29ed2-9c0f-4754-8df3-b43d14726846'(virt4) 2017-10-01 16:01:47,265-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [4e36e836] EVENT_ID: VM_DOWN(61), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM prs is down. 2017-10-01 16:01:47,272-04 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler10) [4e36e836] Failed to invoke scheduled method poll: null 2017-10-01 16:01:47,277-04 INFO [org.ovirt.engine.core.bll.ProcessDownVmCommand] (org.ovirt.thread.pool-6-thread-47) [69930290] Running command: ProcessDownVmCommand internal: true. 2017-10-01 16:01:47,292-04 INFO [org.ovirt.engine.core.bll.ProcessDownVmCommand] (org.ovirt.thread.pool-6-thread-47) [7413d709] Running command: ProcessDownVmCommand internal: true. 2017-10-01 16:01:47,307-04 INFO [org.ovirt.engine.core.bll.ProcessDownVmCommand] (org.ovirt.thread.pool-6-thread-47) [17438615] Running command: ProcessDownVmCommand internal: true. 2017-10-01 16:01:47,585-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] FINISH, DestroyVDSCommand, log id: 7acf22e2 2017-10-01 16:01:47,585-04 INFO [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] FINISH, DestroyVmVDSCommand, log id: 2d373104 2017-10-01 16:01:47,589-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] EVENT_ID: USER_STOP_VM(33), Correlation ID: 2bf4b242-4939-4e61-976b-eda93f156af5, Job ID: 770103c5-8785-40bf-bc56-9de4ffb3fad8, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM samba powered off by admin@internal (Host: virt4). 2017-10-01 16:01:47,589-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [7194fe99] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:01:47,589-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler8) [7194fe99] Command 'SpmStatusVDSCommand(HostName = virt2, SpmStatusVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:01:47,589-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler8) [7194fe99] Irs placed on server 'a0ccfd11-3402-4eba-a119-4987561b738e' failed. Proceed Failover 2017-10-01 16:01:47,599-04 INFO [org.ovirt.engine.core.bll.StopVmCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] Running command: StopVmCommand internal: false. Entities affected : ID: dc66c051-24b7-4cb8-99eb-db398d396ef1 Type: VMAction group STOP_VM with role type USER 2017-10-01 16:01:47,614-04 INFO [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] START, DestroyVmVDSCommand( DestroyVmVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vmId='dc66c051-24b7-4cb8-99eb-db398d396ef1', force='false', secondsToWait='0', gracefully='false', reason='', ignoreNoVm='false'}), log id: 36f13bd9 2017-10-01 16:01:47,615-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] START, DestroyVDSCommand(HostName = virt4, DestroyVmVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vmId='dc66c051-24b7-4cb8-99eb-db398d396ef1', force='false', secondsToWait='0', gracefully='false', reason='', ignoreNoVm='false'}), log id: 4a113ecf 2017-10-01 16:01:47,654-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [7194fe99] START, ConnectStoragePoolVDSCommand(HostName = virt1, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vdsId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 60ab6c5c 2017-10-01 16:01:47,654-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [7194fe99] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:01:48,168-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [7194fe99] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt1 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:01:48,168-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [7194fe99] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:01:48,168-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [7194fe99] HostName = virt1 2017-10-01 16:01:48,168-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [7194fe99] Command 'ConnectStoragePoolVDSCommand(HostName = virt1, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vdsId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:01:48,168-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [7194fe99] FINISH, ConnectStoragePoolVDSCommand, log id: 60ab6c5c 2017-10-01 16:01:48,168-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler8) [7194fe99] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:01:48,202-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] FINISH, DestroyVDSCommand, log id: 4a113ecf 2017-10-01 16:01:48,202-04 INFO [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] FINISH, DestroyVmVDSCommand, log id: 36f13bd9 2017-10-01 16:01:48,206-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] EVENT_ID: USER_STOP_VM(33), Correlation ID: 2bf4b242-4939-4e61-976b-eda93f156af5, Job ID: 9790906f-a3c4-45a5-ba8f-ee8615bfd82e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM samba1 powered off by admin@internal (Host: virt4). 2017-10-01 16:01:48,216-04 INFO [org.ovirt.engine.core.bll.StopVmCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] Running command: StopVmCommand internal: false. Entities affected : ID: d56363ad-9cb1-4a70-a33a-ce439b044d7d Type: VMAction group STOP_VM with role type USER 2017-10-01 16:01:48,235-04 INFO [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] START, DestroyVmVDSCommand( DestroyVmVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vmId='d56363ad-9cb1-4a70-a33a-ce439b044d7d', force='false', secondsToWait='0', gracefully='false', reason='', ignoreNoVm='false'}), log id: 1e6940d 2017-10-01 16:01:48,235-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] START, DestroyVDSCommand(HostName = virt4, DestroyVmVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vmId='d56363ad-9cb1-4a70-a33a-ce439b044d7d', force='false', secondsToWait='0', gracefully='false', reason='', ignoreNoVm='false'}), log id: adafd42 2017-10-01 16:01:48,308-04 INFO [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-6) [1997cff2] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: c8653dba-63ec-4447-872c-0ffcc6af9c0f Type: Storage 2017-10-01 16:01:48,330-04 WARN [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-6) [1997cff2] skipping connect and refresh for all hosts, last master 'true', command status 'true' 2017-10-01 16:01:48,339-04 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-6-thread-4) [7194fe99] Finished reconstruct for pool 'aca61eac-ed8a-484b-bbee-3645148f6405'. Clearing event queue 2017-10-01 16:01:49,251-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] FINISH, DestroyVDSCommand, log id: adafd42 2017-10-01 16:01:49,251-04 INFO [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] FINISH, DestroyVmVDSCommand, log id: 1e6940d 2017-10-01 16:01:49,255-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] EVENT_ID: USER_STOP_VM(33), Correlation ID: 2bf4b242-4939-4e61-976b-eda93f156af5, Job ID: d2970ec2-4b60-4a0e-b000-b92b5ab2adec, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM svn powered off by admin@internal (Host: virt4). 2017-10-01 16:01:49,267-04 INFO [org.ovirt.engine.core.bll.StopVmCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] Running command: StopVmCommand internal: false. Entities affected : ID: 1c7e966f-29e8-4788-b64c-86e52f415061 Type: VMAction group STOP_VM with role type USER 2017-10-01 16:01:49,283-04 INFO [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] START, DestroyVmVDSCommand( DestroyVmVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vmId='1c7e966f-29e8-4788-b64c-86e52f415061', force='false', secondsToWait='0', gracefully='false', reason='', ignoreNoVm='false'}), log id: 75c2f35d 2017-10-01 16:01:49,283-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] START, DestroyVDSCommand(HostName = virt4, DestroyVmVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vmId='1c7e966f-29e8-4788-b64c-86e52f415061', force='false', secondsToWait='0', gracefully='false', reason='', ignoreNoVm='false'}), log id: e79e814 2017-10-01 16:01:49,284-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmsStatisticsFetcher] (DefaultQuartzScheduler4) [2896db35] Fetched 0 VMs from VDS 'e1e71e9d-a0cd-47db-96c6-f724f6cea736' 2017-10-01 16:01:49,284-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler4) [2896db35] VM 'c64d80f9-a3bd-44f3-89c4-f1b9b509cdde'(red) is running in db and not running on VDS 'e1e71e9d-a0cd-47db-96c6-f724f6cea736'(virt3) 2017-10-01 16:01:49,305-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [2896db35] EVENT_ID: VM_DOWN(61), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM red is down. 2017-10-01 16:01:49,306-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler4) [2896db35] VM 'cc1dbe93-bb80-451e-8b3b-c9cc5e90432f'(license) is running in db and not running on VDS 'e1e71e9d-a0cd-47db-96c6-f724f6cea736'(virt3) 2017-10-01 16:01:49,327-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [2896db35] EVENT_ID: VM_DOWN(61), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM license is down. 2017-10-01 16:01:49,330-04 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler4) [2896db35] Failed to invoke scheduled method poll: null 2017-10-01 16:01:49,336-04 INFO [org.ovirt.engine.core.bll.ProcessDownVmCommand] (org.ovirt.thread.pool-6-thread-38) [669ec8a2] Running command: ProcessDownVmCommand internal: true. 2017-10-01 16:01:49,350-04 INFO [org.ovirt.engine.core.bll.ProcessDownVmCommand] (org.ovirt.thread.pool-6-thread-38) [618e83bf] Running command: ProcessDownVmCommand internal: true. 2017-10-01 16:01:49,824-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] FINISH, DestroyVDSCommand, log id: e79e814 2017-10-01 16:01:49,824-04 INFO [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] FINISH, DestroyVmVDSCommand, log id: 75c2f35d 2017-10-01 16:01:49,829-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] EVENT_ID: USER_STOP_VM(33), Correlation ID: 2bf4b242-4939-4e61-976b-eda93f156af5, Job ID: 92d823e7-2390-4c02-8165-584763ecffb6, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM udb powered off by admin@internal (Host: virt4). 2017-10-01 16:01:49,840-04 INFO [org.ovirt.engine.core.bll.StopVmCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] Running command: StopVmCommand internal: false. Entities affected : ID: 59287796-c634-4f3c-94b1-2c9699986637 Type: VMAction group STOP_VM with role type USER 2017-10-01 16:01:49,856-04 INFO [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] START, DestroyVmVDSCommand( DestroyVmVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vmId='59287796-c634-4f3c-94b1-2c9699986637', force='false', secondsToWait='0', gracefully='false', reason='', ignoreNoVm='false'}), log id: a757fd2 2017-10-01 16:01:49,856-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] START, DestroyVDSCommand(HostName = virt1, DestroyVmVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vmId='59287796-c634-4f3c-94b1-2c9699986637', force='false', secondsToWait='0', gracefully='false', reason='', ignoreNoVm='false'}), log id: 368887ff 2017-10-01 16:01:50,507-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] FINISH, DestroyVDSCommand, log id: 368887ff 2017-10-01 16:01:50,507-04 INFO [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] FINISH, DestroyVmVDSCommand, log id: a757fd2 2017-10-01 16:01:50,511-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] EVENT_ID: USER_STOP_VM(33), Correlation ID: 2bf4b242-4939-4e61-976b-eda93f156af5, Job ID: d00b2d39-32be-42fb-9983-41caf744d354, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM vpn1 powered off by admin@internal (Host: virt1). 2017-10-01 16:01:50,523-04 INFO [org.ovirt.engine.core.bll.StopVmCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] Running command: StopVmCommand internal: false. Entities affected : ID: ae65a104-ed3f-40e0-ba92-5708cf338918 Type: VMAction group STOP_VM with role type USER 2017-10-01 16:01:50,538-04 INFO [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] START, DestroyVmVDSCommand( DestroyVmVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vmId='ae65a104-ed3f-40e0-ba92-5708cf338918', force='false', secondsToWait='0', gracefully='false', reason='', ignoreNoVm='false'}), log id: 66856dde 2017-10-01 16:01:50,538-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] START, DestroyVDSCommand(HostName = virt4, DestroyVmVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vmId='ae65a104-ed3f-40e0-ba92-5708cf338918', force='false', secondsToWait='0', gracefully='false', reason='', ignoreNoVm='false'}), log id: 88bb4c9 2017-10-01 16:01:50,791-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmsStatisticsFetcher] (DefaultQuartzScheduler10) [539bfc4f] Fetched 1 VMs from VDS '45f8b331-842e-48e7-9df8-56adddb93836' 2017-10-01 16:01:50,791-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler10) [539bfc4f] VM '899e3a1c-dcc4-4426-9dd4-0f1e8b94a2b8'(ipa) is running in db and not running on VDS '45f8b331-842e-48e7-9df8-56adddb93836'(virt1) 2017-10-01 16:01:50,814-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [539bfc4f] EVENT_ID: VM_DOWN(61), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM ipa is down. 2017-10-01 16:01:50,814-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler10) [539bfc4f] VM '77766cb4-0625-4432-a16f-def5e702102a'(labtest1) is running in db and not running on VDS '45f8b331-842e-48e7-9df8-56adddb93836'(virt1) 2017-10-01 16:01:50,834-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [539bfc4f] EVENT_ID: VM_DOWN(61), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM labtest1 is down. 2017-10-01 16:01:50,835-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler10) [539bfc4f] VM '59287796-c634-4f3c-94b1-2c9699986637'(vpn1) is running in db and not running on VDS '45f8b331-842e-48e7-9df8-56adddb93836'(virt1) 2017-10-01 16:01:50,855-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [539bfc4f] EVENT_ID: VM_DOWN(61), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM vpn1 is down. 2017-10-01 16:01:50,859-04 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler10) [539bfc4f] Failed to invoke scheduled method poll: null 2017-10-01 16:01:50,864-04 INFO [org.ovirt.engine.core.bll.ProcessDownVmCommand] (org.ovirt.thread.pool-6-thread-7) [600c7c7a] Running command: ProcessDownVmCommand internal: true. 2017-10-01 16:01:50,878-04 INFO [org.ovirt.engine.core.bll.ProcessDownVmCommand] (org.ovirt.thread.pool-6-thread-7) [320fbd3c] Running command: ProcessDownVmCommand internal: true. 2017-10-01 16:01:50,892-04 INFO [org.ovirt.engine.core.bll.ProcessDownVmCommand] (org.ovirt.thread.pool-6-thread-7) [1bf48d2c] Running command: ProcessDownVmCommand internal: true. 2017-10-01 16:01:51,089-04 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler7) [366abaea] Storage Domain '63d680c7-4c94-4b29-a6cb-f288a128b9bc:iso' was reported by Host 'virt4' as Active in Pool 'aca61eac-ed8a-484b-bbee-3645148f6405', moving to active status 2017-10-01 16:01:51,096-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] FINISH, DestroyVDSCommand, log id: 88bb4c9 2017-10-01 16:01:51,096-04 INFO [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] FINISH, DestroyVmVDSCommand, log id: 66856dde 2017-10-01 16:01:51,106-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] EVENT_ID: USER_STOP_VM(33), Correlation ID: 2bf4b242-4939-4e61-976b-eda93f156af5, Job ID: 0bf41032-4c0a-4e2b-9527-a8f817c41073, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM vpn2 powered off by admin@internal (Host: virt4). 2017-10-01 16:01:51,116-04 INFO [org.ovirt.engine.core.bll.StopVmCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] Running command: StopVmCommand internal: false. Entities affected : ID: f87ec08d-0fdf-42ba-8aac-c66c404d7aae Type: VMAction group STOP_VM with role type USER 2017-10-01 16:01:51,131-04 INFO [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] START, DestroyVmVDSCommand( DestroyVmVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vmId='f87ec08d-0fdf-42ba-8aac-c66c404d7aae', force='false', secondsToWait='0', gracefully='false', reason='', ignoreNoVm='false'}), log id: 617018aa 2017-10-01 16:01:51,132-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] START, DestroyVDSCommand(HostName = virt4, DestroyVmVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vmId='f87ec08d-0fdf-42ba-8aac-c66c404d7aae', force='false', secondsToWait='0', gracefully='false', reason='', ignoreNoVm='false'}), log id: 63094a9e 2017-10-01 16:01:51,134-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmsStatisticsFetcher] (DefaultQuartzScheduler2) [784fa946] Fetched 1 VMs from VDS 'a0ccfd11-3402-4eba-a119-4987561b738e' 2017-10-01 16:01:51,135-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler2) [784fa946] VM 'db3e08fb-c111-442d-aeec-1507da30f486'(print) is running in db and not running on VDS 'a0ccfd11-3402-4eba-a119-4987561b738e'(virt2) 2017-10-01 16:01:51,205-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [784fa946] EVENT_ID: VM_DOWN(61), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM print is down. 2017-10-01 16:01:51,206-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler2) [784fa946] VM 'ae56c31c-71f7-433a-b198-a1c2c846325e'(forum) is running in db and not running on VDS 'a0ccfd11-3402-4eba-a119-4987561b738e'(virt2) 2017-10-01 16:01:51,228-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [784fa946] EVENT_ID: VM_DOWN(61), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM forum is down. 2017-10-01 16:01:51,228-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler2) [784fa946] VM '1325cebf-834e-4571-a3e1-82beb0021ffe'(dns1) is running in db and not running on VDS 'a0ccfd11-3402-4eba-a119-4987561b738e'(virt2) 2017-10-01 16:01:51,250-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [784fa946] EVENT_ID: VM_DOWN(61), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM dns1 is down. 2017-10-01 16:01:51,250-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler2) [784fa946] VM 'bfe9a524-2238-4bab-a91f-5dc6ed270896'(pcmode) is running in db and not running on VDS 'a0ccfd11-3402-4eba-a119-4987561b738e'(virt2) 2017-10-01 16:01:51,272-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [784fa946] EVENT_ID: VM_DOWN(61), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM pcmode is down. 2017-10-01 16:01:51,273-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler2) [784fa946] VM 'a82b4cc3-6ef6-4a8e-97ef-1d4ea1b7e07c'(indigo) is running in db and not running on VDS 'a0ccfd11-3402-4eba-a119-4987561b738e'(virt2) 2017-10-01 16:01:51,295-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [784fa946] EVENT_ID: VM_DOWN(61), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM indigo is down. 2017-10-01 16:01:51,295-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler2) [784fa946] VM 'a8d4e149-55ec-4961-a414-d7d1e1b05a59'(fix) is running in db and not running on VDS 'a0ccfd11-3402-4eba-a119-4987561b738e'(virt2) 2017-10-01 16:01:51,317-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [784fa946] EVENT_ID: VM_DOWN(61), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM fix is down. 2017-10-01 16:01:51,317-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler2) [784fa946] VM 'aa487207-7ff4-465a-9d9b-2a103d50dc77'(ftp1) is running in db and not running on VDS 'a0ccfd11-3402-4eba-a119-4987561b738e'(virt2) 2017-10-01 16:01:51,338-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [784fa946] EVENT_ID: VM_DOWN(61), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM ftp1 is down. 2017-10-01 16:01:51,338-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler2) [784fa946] VM '4ba80442-b361-4026-ae8c-895fd28042b0'(dist) is running in db and not running on VDS 'a0ccfd11-3402-4eba-a119-4987561b738e'(virt2) 2017-10-01 16:01:51,359-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [784fa946] EVENT_ID: VM_DOWN(61), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM dist is down. 2017-10-01 16:01:51,369-04 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler2) [784fa946] Failed to invoke scheduled method poll: null 2017-10-01 16:01:51,374-04 INFO [org.ovirt.engine.core.bll.ProcessDownVmCommand] (org.ovirt.thread.pool-6-thread-18) [495f4944] Running command: ProcessDownVmCommand internal: true. 2017-10-01 16:01:51,392-04 INFO [org.ovirt.engine.core.bll.ProcessDownVmCommand] (org.ovirt.thread.pool-6-thread-18) [3bae20ea] Running command: ProcessDownVmCommand internal: true. 2017-10-01 16:01:51,406-04 INFO [org.ovirt.engine.core.bll.ProcessDownVmCommand] (org.ovirt.thread.pool-6-thread-18) [10bda7fc] Running command: ProcessDownVmCommand internal: true. 2017-10-01 16:01:51,421-04 INFO [org.ovirt.engine.core.bll.ProcessDownVmCommand] (org.ovirt.thread.pool-6-thread-18) [2bba6740] Running command: ProcessDownVmCommand internal: true. 2017-10-01 16:01:51,435-04 INFO [org.ovirt.engine.core.bll.ProcessDownVmCommand] (org.ovirt.thread.pool-6-thread-18) [12c20d92] Running command: ProcessDownVmCommand internal: true. 2017-10-01 16:01:51,449-04 INFO [org.ovirt.engine.core.bll.ProcessDownVmCommand] (org.ovirt.thread.pool-6-thread-18) [af65db2] Running command: ProcessDownVmCommand internal: true. 2017-10-01 16:01:51,462-04 INFO [org.ovirt.engine.core.bll.ProcessDownVmCommand] (org.ovirt.thread.pool-6-thread-18) [1fdba455] Running command: ProcessDownVmCommand internal: true. 2017-10-01 16:01:51,475-04 INFO [org.ovirt.engine.core.bll.ProcessDownVmCommand] (org.ovirt.thread.pool-6-thread-18) [59dfd604] Running command: ProcessDownVmCommand internal: true. 2017-10-01 16:01:51,678-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] FINISH, DestroyVDSCommand, log id: 63094a9e 2017-10-01 16:01:51,678-04 INFO [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] FINISH, DestroyVmVDSCommand, log id: 617018aa 2017-10-01 16:01:51,682-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] EVENT_ID: USER_STOP_VM(33), Correlation ID: 2bf4b242-4939-4e61-976b-eda93f156af5, Job ID: 756370ea-2b5e-4b35-a193-9952c4a369d1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM webapp powered off by admin@internal (Host: virt4). 2017-10-01 16:01:51,693-04 INFO [org.ovirt.engine.core.bll.StopVmCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] Running command: StopVmCommand internal: false. Entities affected : ID: 8a622713-85a4-4ff3-85c2-9c0db9aa0925 Type: VMAction group STOP_VM with role type USER 2017-10-01 16:01:51,698-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] EVENT_ID: VM_ALREADY_IN_REQUESTED_STATUS(254), Correlation ID: 2bf4b242-4939-4e61-976b-eda93f156af5, Job ID: f14417af-b5e9-4291-bf96-bb533cabe894, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM webapp1 is already Down, Stop VM was skipped. User: admin@internal. 2017-10-01 16:01:51,708-04 INFO [org.ovirt.engine.core.bll.StopVmCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] Running command: StopVmCommand internal: false. Entities affected : ID: 64734fca-4d6c-443c-82d1-abf7cf106601 Type: VMAction group STOP_VM with role type USER 2017-10-01 16:01:51,724-04 INFO [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] START, DestroyVmVDSCommand( DestroyVmVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vmId='64734fca-4d6c-443c-82d1-abf7cf106601', force='false', secondsToWait='0', gracefully='false', reason='', ignoreNoVm='false'}), log id: 637f045e 2017-10-01 16:01:51,724-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] START, DestroyVDSCommand(HostName = virt1, DestroyVmVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vmId='64734fca-4d6c-443c-82d1-abf7cf106601', force='false', secondsToWait='0', gracefully='false', reason='', ignoreNoVm='false'}), log id: 489f3bb5 2017-10-01 16:01:53,269-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] FINISH, DestroyVDSCommand, log id: 489f3bb5 2017-10-01 16:01:53,269-04 INFO [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] FINISH, DestroyVmVDSCommand, log id: 637f045e 2017-10-01 16:01:53,306-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] EVENT_ID: USER_STOP_VM(33), Correlation ID: 2bf4b242-4939-4e61-976b-eda93f156af5, Job ID: 5297af33-6e54-49ea-b33d-bd2f521e0f33, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM webdav powered off by admin@internal (Host: virt1). 2017-10-01 16:01:53,317-04 INFO [org.ovirt.engine.core.bll.StopVmCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] Running command: StopVmCommand internal: false. Entities affected : ID: 1734fed5-fc97-4714-bb63-55e8f4eb6d34 Type: VMAction group STOP_VM with role type USER 2017-10-01 16:01:53,322-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] EVENT_ID: VM_ALREADY_IN_REQUESTED_STATUS(254), Correlation ID: 2bf4b242-4939-4e61-976b-eda93f156af5, Job ID: e108510f-fb9b-4964-8faf-eb32afa32e26, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM wiki is already Down, Stop VM was skipped. User: admin@internal. 2017-10-01 16:01:53,333-04 INFO [org.ovirt.engine.core.bll.StopVmCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] Running command: StopVmCommand internal: false. Entities affected : ID: fffde246-af07-480d-819c-5614c5ddf5ec Type: VMAction group STOP_VM with role type USER 2017-10-01 16:01:53,337-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] EVENT_ID: VM_ALREADY_IN_REQUESTED_STATUS(254), Correlation ID: 2bf4b242-4939-4e61-976b-eda93f156af5, Job ID: 19b19557-9f7b-476b-881c-62e6ac8a66f7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM win7test is already Down, Stop VM was skipped. User: admin@internal. 2017-10-01 16:01:53,348-04 INFO [org.ovirt.engine.core.bll.StopVmCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] Running command: StopVmCommand internal: false. Entities affected : ID: 3e5a5728-8940-4fe7-b4af-11fb8e12f116 Type: VMAction group STOP_VM with role type USER 2017-10-01 16:01:53,364-04 INFO [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] START, DestroyVmVDSCommand( DestroyVmVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vmId='3e5a5728-8940-4fe7-b4af-11fb8e12f116', force='false', secondsToWait='0', gracefully='false', reason='', ignoreNoVm='false'}), log id: 64f88d17 2017-10-01 16:01:53,364-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] START, DestroyVDSCommand(HostName = virt4, DestroyVmVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vmId='3e5a5728-8940-4fe7-b4af-11fb8e12f116', force='false', secondsToWait='0', gracefully='false', reason='', ignoreNoVm='false'}), log id: 35f17c4 2017-10-01 16:01:53,906-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] FINISH, DestroyVDSCommand, log id: 35f17c4 2017-10-01 16:01:53,906-04 INFO [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] FINISH, DestroyVmVDSCommand, log id: 64f88d17 2017-10-01 16:01:53,910-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] EVENT_ID: USER_STOP_VM(33), Correlation ID: 2bf4b242-4939-4e61-976b-eda93f156af5, Job ID: 1e2f9a47-2309-4025-be86-6026f105c55f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM www powered off by admin@internal (Host: virt4). 2017-10-01 16:01:53,919-04 INFO [org.ovirt.engine.core.bll.StopVmCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] Running command: StopVmCommand internal: false. Entities affected : ID: d66235b4-7c8d-4da2-a572-82ecc9c53a59 Type: VMAction group STOP_VM with role type USER 2017-10-01 16:01:53,934-04 INFO [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] START, DestroyVmVDSCommand( DestroyVmVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vmId='d66235b4-7c8d-4da2-a572-82ecc9c53a59', force='false', secondsToWait='0', gracefully='false', reason='', ignoreNoVm='false'}), log id: 57b0e67 2017-10-01 16:01:53,934-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] START, DestroyVDSCommand(HostName = virt4, DestroyVmVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vmId='d66235b4-7c8d-4da2-a572-82ecc9c53a59', force='false', secondsToWait='0', gracefully='false', reason='', ignoreNoVm='false'}), log id: 63958b43 2017-10-01 16:01:55,516-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] FINISH, DestroyVDSCommand, log id: 63958b43 2017-10-01 16:01:55,516-04 INFO [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] FINISH, DestroyVmVDSCommand, log id: 57b0e67 2017-10-01 16:01:55,521-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] EVENT_ID: USER_STOP_VM(33), Correlation ID: 2bf4b242-4939-4e61-976b-eda93f156af5, Job ID: f57143fa-074e-4a81-a6c1-c51c3b22ecd2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM www-las powered off by admin@internal (Host: virt4). 2017-10-01 16:01:55,533-04 INFO [org.ovirt.engine.core.bll.StopVmCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] Running command: StopVmCommand internal: false. Entities affected : ID: 3bb83ad5-dc00-445d-8708-2db18c53e0e6 Type: VMAction group STOP_VM with role type USER 2017-10-01 16:01:55,548-04 INFO [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] START, DestroyVmVDSCommand( DestroyVmVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vmId='3bb83ad5-dc00-445d-8708-2db18c53e0e6', force='false', secondsToWait='0', gracefully='false', reason='', ignoreNoVm='false'}), log id: 49953d79 2017-10-01 16:01:55,549-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] START, DestroyVDSCommand(HostName = virt2, DestroyVmVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vmId='3bb83ad5-dc00-445d-8708-2db18c53e0e6', force='false', secondsToWait='0', gracefully='false', reason='', ignoreNoVm='false'}), log id: 44a16d46 2017-10-01 16:01:56,104-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] FINISH, DestroyVDSCommand, log id: 44a16d46 2017-10-01 16:01:56,104-04 INFO [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] FINISH, DestroyVmVDSCommand, log id: 49953d79 2017-10-01 16:01:56,109-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] EVENT_ID: USER_STOP_VM(33), Correlation ID: 2bf4b242-4939-4e61-976b-eda93f156af5, Job ID: ebb7bdd9-0d77-4a8b-8171-4c1bede482b4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM www-vhost powered off by admin@internal (Host: virt2). 2017-10-01 16:01:56,120-04 INFO [org.ovirt.engine.core.bll.StopVmCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] Running command: StopVmCommand internal: false. Entities affected : ID: 8391f9a7-7fe7-4025-8fba-6273ae587a52 Type: VMAction group STOP_VM with role type USER 2017-10-01 16:01:56,136-04 INFO [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] START, DestroyVmVDSCommand( DestroyVmVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vmId='8391f9a7-7fe7-4025-8fba-6273ae587a52', force='false', secondsToWait='0', gracefully='false', reason='', ignoreNoVm='false'}), log id: dc6dba8 2017-10-01 16:01:56,136-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] START, DestroyVDSCommand(HostName = virt4, DestroyVmVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vmId='8391f9a7-7fe7-4025-8fba-6273ae587a52', force='false', secondsToWait='0', gracefully='false', reason='', ignoreNoVm='false'}), log id: a0837c9 2017-10-01 16:01:56,230-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [50417af6] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 16:01:56,669-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] FINISH, DestroyVDSCommand, log id: a0837c9 2017-10-01 16:01:56,669-04 INFO [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] FINISH, DestroyVmVDSCommand, log id: dc6dba8 2017-10-01 16:01:56,674-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-32) [2bf4b242-4939-4e61-976b-eda93f156af5] EVENT_ID: USER_STOP_VM(33), Correlation ID: 2bf4b242-4939-4e61-976b-eda93f156af5, Job ID: 1b1d856c-77db-4f54-90ce-1fb7150dc17a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM archive powered off by admin@internal (Host: virt4). 2017-10-01 16:01:59,298-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [ac79afd] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt1 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:01:59,299-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler1) [ac79afd] Command 'SpmStatusVDSCommand(HostName = virt1, SpmStatusVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:02:00,309-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [ac79afd] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt1 command HSMGetAllTasksStatusesVDS failed: Not SPM 2017-10-01 16:02:00,309-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler1) [ac79afd] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand' return value ' 2017-10-01 16:02:00,309-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler1) [ac79afd] HostName = virt1 2017-10-01 16:02:00,309-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler1) [ac79afd] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = virt1, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836'})' execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM 2017-10-01 16:02:00,440-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [ac79afd] START, ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 470ef2c1 2017-10-01 16:02:00,440-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [ac79afd] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:02:01,526-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [ac79afd] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:02:01,526-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [ac79afd] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:02:01,526-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [ac79afd] HostName = virt4 2017-10-01 16:02:01,526-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [ac79afd] Command 'ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:02:01,526-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [ac79afd] FINISH, ConnectStoragePoolVDSCommand, log id: 470ef2c1 2017-10-01 16:02:01,526-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler1) [ac79afd] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:02:01,847-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [ac79afd] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:02:01,847-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler1) [ac79afd] Command 'SpmStatusVDSCommand(HostName = virt4, SpmStatusVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:02:01,847-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler1) [ac79afd] Irs placed on server '5ae29ed2-9c0f-4754-8df3-b43d14726846' failed. Proceed Failover 2017-10-01 16:02:01,916-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [ac79afd] START, ConnectStoragePoolVDSCommand(HostName = virt1, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vdsId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 174df444 2017-10-01 16:02:01,916-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [ac79afd] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:02:01,929-04 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-14) [] domain '63d680c7-4c94-4b29-a6cb-f288a128b9bc:iso' in problem 'PROBLEMATIC'. vds: 'virt1' 2017-10-01 16:02:02,278-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [ac79afd] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt1 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:02:02,278-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [ac79afd] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:02:02,278-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [ac79afd] HostName = virt1 2017-10-01 16:02:02,278-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [ac79afd] Command 'ConnectStoragePoolVDSCommand(HostName = virt1, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vdsId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:02:02,278-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [ac79afd] FINISH, ConnectStoragePoolVDSCommand, log id: 174df444 2017-10-01 16:02:02,278-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler1) [ac79afd] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:02:02,390-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmsStatisticsFetcher] (DefaultQuartzScheduler2) [1fef106b] Fetched 0 VMs from VDS '5ae29ed2-9c0f-4754-8df3-b43d14726846' 2017-10-01 16:02:02,390-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler2) [1fef106b] VM '07331c78-6e09-41b4-9b29-edda32f46351'(samba) is running in db and not running on VDS '5ae29ed2-9c0f-4754-8df3-b43d14726846'(virt4) 2017-10-01 16:02:02,459-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [1fef106b] EVENT_ID: VM_DOWN(61), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM samba is down. 2017-10-01 16:02:02,460-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler2) [1fef106b] VM 'dc66c051-24b7-4cb8-99eb-db398d396ef1'(samba1) is running in db and not running on VDS '5ae29ed2-9c0f-4754-8df3-b43d14726846'(virt4) 2017-10-01 16:02:02,481-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [1fef106b] EVENT_ID: VM_DOWN(61), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM samba1 is down. 2017-10-01 16:02:02,482-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler2) [1fef106b] VM '3e5a5728-8940-4fe7-b4af-11fb8e12f116'(www) is running in db and not running on VDS '5ae29ed2-9c0f-4754-8df3-b43d14726846'(virt4) 2017-10-01 16:02:02,502-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [1fef106b] EVENT_ID: VM_DOWN(61), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM www is down. 2017-10-01 16:02:02,502-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler2) [1fef106b] VM 'f87ec08d-0fdf-42ba-8aac-c66c404d7aae'(webapp) is running in db and not running on VDS '5ae29ed2-9c0f-4754-8df3-b43d14726846'(virt4) 2017-10-01 16:02:02,523-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [1fef106b] EVENT_ID: VM_DOWN(61), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM webapp is down. 2017-10-01 16:02:02,523-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler2) [1fef106b] VM '8391f9a7-7fe7-4025-8fba-6273ae587a52'(archive) is running in db and not running on VDS '5ae29ed2-9c0f-4754-8df3-b43d14726846'(virt4) 2017-10-01 16:02:02,544-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [1fef106b] EVENT_ID: VM_DOWN(61), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM archive is down. 2017-10-01 16:02:02,544-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler2) [1fef106b] VM '1c7e966f-29e8-4788-b64c-86e52f415061'(udb) is running in db and not running on VDS '5ae29ed2-9c0f-4754-8df3-b43d14726846'(virt4) 2017-10-01 16:02:02,565-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [1fef106b] EVENT_ID: VM_DOWN(61), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM udb is down. 2017-10-01 16:02:02,565-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler2) [1fef106b] VM '9ea2821e-557f-45cb-9872-1e6c5aeaf246'(rs) is running in db and not running on VDS '5ae29ed2-9c0f-4754-8df3-b43d14726846'(virt4) 2017-10-01 16:02:02,586-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [1fef106b] EVENT_ID: VM_DOWN(61), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM rs is down. 2017-10-01 16:02:02,586-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler2) [1fef106b] VM 'd66235b4-7c8d-4da2-a572-82ecc9c53a59'(www-las) is running in db and not running on VDS '5ae29ed2-9c0f-4754-8df3-b43d14726846'(virt4) 2017-10-01 16:02:02,607-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [1fef106b] EVENT_ID: VM_DOWN(61), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM www-las is down. 2017-10-01 16:02:02,607-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler2) [1fef106b] VM 'd56363ad-9cb1-4a70-a33a-ce439b044d7d'(svn) is running in db and not running on VDS '5ae29ed2-9c0f-4754-8df3-b43d14726846'(virt4) 2017-10-01 16:02:02,628-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [1fef106b] EVENT_ID: VM_DOWN(61), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM svn is down. 2017-10-01 16:02:02,628-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler2) [1fef106b] VM 'ae65a104-ed3f-40e0-ba92-5708cf338918'(vpn2) is running in db and not running on VDS '5ae29ed2-9c0f-4754-8df3-b43d14726846'(virt4) 2017-10-01 16:02:02,649-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [1fef106b] EVENT_ID: VM_DOWN(61), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM vpn2 is down. 2017-10-01 16:02:02,656-04 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler2) [1fef106b] Failed to invoke scheduled method poll: null 2017-10-01 16:02:02,661-04 INFO [org.ovirt.engine.core.bll.ProcessDownVmCommand] (org.ovirt.thread.pool-6-thread-49) [142a23c9] Running command: ProcessDownVmCommand internal: true. 2017-10-01 16:02:02,676-04 INFO [org.ovirt.engine.core.bll.ProcessDownVmCommand] (org.ovirt.thread.pool-6-thread-49) [73bd539e] Running command: ProcessDownVmCommand internal: true. 2017-10-01 16:02:02,690-04 INFO [org.ovirt.engine.core.bll.ProcessDownVmCommand] (org.ovirt.thread.pool-6-thread-49) [3c3229f4] Running command: ProcessDownVmCommand internal: true. 2017-10-01 16:02:02,705-04 INFO [org.ovirt.engine.core.bll.ProcessDownVmCommand] (org.ovirt.thread.pool-6-thread-49) [54183592] Running command: ProcessDownVmCommand internal: true. 2017-10-01 16:02:02,719-04 INFO [org.ovirt.engine.core.bll.ProcessDownVmCommand] (org.ovirt.thread.pool-6-thread-49) [25701389] Running command: ProcessDownVmCommand internal: true. 2017-10-01 16:02:02,732-04 INFO [org.ovirt.engine.core.bll.ProcessDownVmCommand] (org.ovirt.thread.pool-6-thread-49) [6aea90f0] Running command: ProcessDownVmCommand internal: true. 2017-10-01 16:02:02,745-04 INFO [org.ovirt.engine.core.bll.ProcessDownVmCommand] (org.ovirt.thread.pool-6-thread-49) [7c5bbc25] Running command: ProcessDownVmCommand internal: true. 2017-10-01 16:02:02,758-04 INFO [org.ovirt.engine.core.bll.ProcessDownVmCommand] (org.ovirt.thread.pool-6-thread-49) [7ee29018] Running command: ProcessDownVmCommand internal: true. 2017-10-01 16:02:02,771-04 INFO [org.ovirt.engine.core.bll.ProcessDownVmCommand] (org.ovirt.thread.pool-6-thread-49) [5cc43eff] Running command: ProcessDownVmCommand internal: true. 2017-10-01 16:02:02,784-04 INFO [org.ovirt.engine.core.bll.ProcessDownVmCommand] (org.ovirt.thread.pool-6-thread-49) [798c1f8a] Running command: ProcessDownVmCommand internal: true. 2017-10-01 16:02:03,399-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [ac79afd] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt1 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:02:03,399-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler1) [ac79afd] Command 'SpmStatusVDSCommand(HostName = virt1, SpmStatusVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:02:03,399-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler1) [ac79afd] Irs placed on server '45f8b331-842e-48e7-9df8-56adddb93836' failed. Proceed Failover 2017-10-01 16:02:03,456-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [ac79afd] START, ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 3fc7b31e 2017-10-01 16:02:03,456-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [ac79afd] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:02:04,336-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [ac79afd] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:02:04,336-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [ac79afd] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:02:04,336-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [ac79afd] HostName = virt2 2017-10-01 16:02:04,336-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [ac79afd] Command 'ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:02:04,336-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [ac79afd] FINISH, ConnectStoragePoolVDSCommand, log id: 3fc7b31e 2017-10-01 16:02:04,336-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler1) [ac79afd] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:02:05,456-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [ac79afd] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:02:05,456-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler1) [ac79afd] Command 'SpmStatusVDSCommand(HostName = virt2, SpmStatusVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:02:05,456-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler1) [ac79afd] Irs placed on server 'a0ccfd11-3402-4eba-a119-4987561b738e' failed. Proceed Failover 2017-10-01 16:02:05,513-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [ac79afd] START, ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 3f2cb9d 2017-10-01 16:02:05,513-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [ac79afd] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:02:05,864-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [ac79afd] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:02:05,864-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [ac79afd] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:02:05,864-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [ac79afd] HostName = virt3 2017-10-01 16:02:05,864-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [ac79afd] Command 'ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:02:05,864-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [ac79afd] FINISH, ConnectStoragePoolVDSCommand, log id: 3f2cb9d 2017-10-01 16:02:05,864-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler1) [ac79afd] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:02:05,994-04 INFO [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-41) [5cd23d5f] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: c8653dba-63ec-4447-872c-0ffcc6af9c0f Type: Storage 2017-10-01 16:02:06,007-04 WARN [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-41) [5cd23d5f] skipping connect and refresh for all hosts, last master 'true', command status 'true' 2017-10-01 16:02:06,024-04 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-6-thread-40) [ac79afd] Finished reconstruct for pool 'aca61eac-ed8a-484b-bbee-3645148f6405'. Clearing event queue 2017-10-01 16:02:06,206-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmsStatisticsFetcher] (DefaultQuartzScheduler5) [171f84bd] Fetched 0 VMs from VDS '45f8b331-842e-48e7-9df8-56adddb93836' 2017-10-01 16:02:06,206-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler5) [171f84bd] VM '64734fca-4d6c-443c-82d1-abf7cf106601'(webdav) is running in db and not running on VDS '45f8b331-842e-48e7-9df8-56adddb93836'(virt1) 2017-10-01 16:02:06,277-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [171f84bd] EVENT_ID: VM_DOWN(61), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM webdav is down. 2017-10-01 16:02:06,281-04 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler5) [171f84bd] Failed to invoke scheduled method poll: null 2017-10-01 16:02:06,286-04 INFO [org.ovirt.engine.core.bll.ProcessDownVmCommand] (org.ovirt.thread.pool-6-thread-44) [15c217b7] Running command: ProcessDownVmCommand internal: true. 2017-10-01 16:02:06,764-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmsStatisticsFetcher] (DefaultQuartzScheduler3) [2f85cdbf] Fetched 0 VMs from VDS 'a0ccfd11-3402-4eba-a119-4987561b738e' 2017-10-01 16:02:06,764-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler3) [2f85cdbf] VM '3bb83ad5-dc00-445d-8708-2db18c53e0e6'(www-vhost) is running in db and not running on VDS 'a0ccfd11-3402-4eba-a119-4987561b738e'(virt2) 2017-10-01 16:02:06,786-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [2f85cdbf] EVENT_ID: VM_DOWN(61), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM www-vhost is down. 2017-10-01 16:02:06,789-04 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler3) [2f85cdbf] Failed to invoke scheduled method poll: null 2017-10-01 16:02:06,794-04 INFO [org.ovirt.engine.core.bll.ProcessDownVmCommand] (org.ovirt.thread.pool-6-thread-9) [8f2b8dd] Running command: ProcessDownVmCommand internal: true. 2017-10-01 16:02:17,011-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [2dcab641] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command HSMGetAllTasksStatusesVDS failed: Not SPM: () 2017-10-01 16:02:17,011-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler5) [2dcab641] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand' return value ' 2017-10-01 16:02:17,011-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler5) [2dcab641] HostName = virt3 2017-10-01 16:02:17,011-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler5) [2dcab641] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = virt3, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736'})' execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () 2017-10-01 16:02:17,075-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [2dcab641] START, ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 49b1fc07 2017-10-01 16:02:17,075-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [2dcab641] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:02:17,662-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [2dcab641] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:02:17,662-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [2dcab641] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:02:17,662-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [2dcab641] HostName = virt2 2017-10-01 16:02:17,662-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [2dcab641] Command 'ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:02:17,662-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [2dcab641] FINISH, ConnectStoragePoolVDSCommand, log id: 49b1fc07 2017-10-01 16:02:17,662-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler5) [2dcab641] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:02:18,786-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [2dcab641] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:02:18,786-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler5) [2dcab641] Command 'SpmStatusVDSCommand(HostName = virt2, SpmStatusVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:02:18,786-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler5) [2dcab641] Irs placed on server 'a0ccfd11-3402-4eba-a119-4987561b738e' failed. Proceed Failover 2017-10-01 16:02:18,844-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [2dcab641] START, ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 498c00a0 2017-10-01 16:02:18,844-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [2dcab641] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:02:19,452-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [2dcab641] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:02:19,452-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [2dcab641] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:02:19,452-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [2dcab641] HostName = virt3 2017-10-01 16:02:19,452-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [2dcab641] Command 'ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:02:19,452-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [2dcab641] FINISH, ConnectStoragePoolVDSCommand, log id: 498c00a0 2017-10-01 16:02:19,452-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler5) [2dcab641] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:02:20,565-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler5) [2dcab641] Irs placed on server 'e1e71e9d-a0cd-47db-96c6-f724f6cea736' failed. Proceed Failover 2017-10-01 16:02:20,623-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [2dcab641] START, ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 46034556 2017-10-01 16:02:20,623-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [2dcab641] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:02:21,287-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [2dcab641] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:02:21,287-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [2dcab641] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:02:21,287-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [2dcab641] HostName = virt4 2017-10-01 16:02:21,287-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [2dcab641] Command 'ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:02:21,287-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [2dcab641] FINISH, ConnectStoragePoolVDSCommand, log id: 46034556 2017-10-01 16:02:21,287-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler5) [2dcab641] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:02:21,460-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [2dcab641] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:02:21,460-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler5) [2dcab641] Command 'SpmStatusVDSCommand(HostName = virt4, SpmStatusVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:02:21,460-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler5) [2dcab641] Irs placed on server '5ae29ed2-9c0f-4754-8df3-b43d14726846' failed. Proceed Failover 2017-10-01 16:02:21,518-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [2dcab641] START, ConnectStoragePoolVDSCommand(HostName = virt1, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vdsId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 4f667c85 2017-10-01 16:02:21,518-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [2dcab641] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:02:21,796-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [2dcab641] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt1 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:02:21,796-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [2dcab641] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:02:21,796-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [2dcab641] HostName = virt1 2017-10-01 16:02:21,796-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [2dcab641] Command 'ConnectStoragePoolVDSCommand(HostName = virt1, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vdsId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:02:21,796-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [2dcab641] FINISH, ConnectStoragePoolVDSCommand, log id: 4f667c85 2017-10-01 16:02:21,796-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler5) [2dcab641] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:02:21,923-04 INFO [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-10) [2eb38d83] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: c8653dba-63ec-4447-872c-0ffcc6af9c0f Type: Storage 2017-10-01 16:02:21,933-04 WARN [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-10) [2eb38d83] skipping connect and refresh for all hosts, last master 'true', command status 'true' 2017-10-01 16:02:21,940-04 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-6-thread-5) [2dcab641] Finished reconstruct for pool 'aca61eac-ed8a-484b-bbee-3645148f6405'. Clearing event queue 2017-10-01 16:02:32,118-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [55e37e93] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt1 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:02:32,118-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler1) [55e37e93] Command 'SpmStatusVDSCommand(HostName = virt1, SpmStatusVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:02:32,269-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [55e37e93] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt1 command HSMGetAllTasksStatusesVDS failed: Not SPM 2017-10-01 16:02:32,269-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler1) [55e37e93] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand' return value ' 2017-10-01 16:02:32,269-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler1) [55e37e93] HostName = virt1 2017-10-01 16:02:32,269-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler1) [55e37e93] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = virt1, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836'})' execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM 2017-10-01 16:02:32,393-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [55e37e93] START, ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 16c074c 2017-10-01 16:02:32,394-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [55e37e93] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:02:32,778-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [55e37e93] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:02:32,778-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [55e37e93] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:02:32,778-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [55e37e93] HostName = virt4 2017-10-01 16:02:32,778-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [55e37e93] Command 'ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:02:32,778-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [55e37e93] FINISH, ConnectStoragePoolVDSCommand, log id: 16c074c 2017-10-01 16:02:32,778-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler1) [55e37e93] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:02:33,903-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [55e37e93] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:02:33,903-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler1) [55e37e93] Command 'SpmStatusVDSCommand(HostName = virt4, SpmStatusVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:02:33,903-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler1) [55e37e93] Irs placed on server '5ae29ed2-9c0f-4754-8df3-b43d14726846' failed. Proceed Failover 2017-10-01 16:02:33,972-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [55e37e93] START, ConnectStoragePoolVDSCommand(HostName = virt1, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vdsId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 599099de 2017-10-01 16:02:33,972-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [55e37e93] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:02:34,569-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [55e37e93] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt1 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:02:34,569-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [55e37e93] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:02:34,569-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [55e37e93] HostName = virt1 2017-10-01 16:02:34,569-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [55e37e93] Command 'ConnectStoragePoolVDSCommand(HostName = virt1, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vdsId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:02:34,569-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [55e37e93] FINISH, ConnectStoragePoolVDSCommand, log id: 599099de 2017-10-01 16:02:34,569-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler1) [55e37e93] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:02:35,689-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [55e37e93] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt1 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:02:35,689-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler1) [55e37e93] Command 'SpmStatusVDSCommand(HostName = virt1, SpmStatusVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:02:35,689-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler1) [55e37e93] Irs placed on server '45f8b331-842e-48e7-9df8-56adddb93836' failed. Proceed Failover 2017-10-01 16:02:35,747-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [55e37e93] START, ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 4f02dca3 2017-10-01 16:02:35,747-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [55e37e93] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:02:36,404-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [55e37e93] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:02:36,404-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [55e37e93] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:02:36,404-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [55e37e93] HostName = virt3 2017-10-01 16:02:36,404-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [55e37e93] Command 'ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:02:36,404-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [55e37e93] FINISH, ConnectStoragePoolVDSCommand, log id: 4f02dca3 2017-10-01 16:02:36,404-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler1) [55e37e93] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:02:36,559-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler1) [55e37e93] Irs placed on server 'e1e71e9d-a0cd-47db-96c6-f724f6cea736' failed. Proceed Failover 2017-10-01 16:02:36,616-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [55e37e93] START, ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 5befc746 2017-10-01 16:02:36,616-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [55e37e93] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:02:37,708-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [55e37e93] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:02:37,708-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [55e37e93] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:02:37,708-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [55e37e93] HostName = virt2 2017-10-01 16:02:37,708-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [55e37e93] Command 'ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:02:37,708-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [55e37e93] FINISH, ConnectStoragePoolVDSCommand, log id: 5befc746 2017-10-01 16:02:37,708-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler1) [55e37e93] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:02:37,844-04 INFO [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-39) [9a5e7e] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: c8653dba-63ec-4447-872c-0ffcc6af9c0f Type: Storage 2017-10-01 16:02:37,855-04 WARN [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-39) [9a5e7e] skipping connect and refresh for all hosts, last master 'true', command status 'true' 2017-10-01 16:02:37,868-04 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-6-thread-42) [55e37e93] Finished reconstruct for pool 'aca61eac-ed8a-484b-bbee-3645148f6405'. Clearing event queue 2017-10-01 16:02:47,898-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [4b1c14ef] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:02:47,898-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler5) [4b1c14ef] Command 'SpmStatusVDSCommand(HostName = virt2, SpmStatusVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:02:48,907-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [4b1c14ef] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command HSMGetAllTasksStatusesVDS failed: Not SPM 2017-10-01 16:02:48,907-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler5) [4b1c14ef] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand' return value ' 2017-10-01 16:02:48,907-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler5) [4b1c14ef] HostName = virt2 2017-10-01 16:02:48,907-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler5) [4b1c14ef] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = virt2, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e'})' execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM 2017-10-01 16:02:48,970-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [4b1c14ef] START, ConnectStoragePoolVDSCommand(HostName = virt1, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vdsId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 2c999b9b 2017-10-01 16:02:48,970-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [4b1c14ef] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:02:49,687-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [4b1c14ef] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt1 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:02:49,687-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [4b1c14ef] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:02:49,687-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [4b1c14ef] HostName = virt1 2017-10-01 16:02:49,687-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [4b1c14ef] Command 'ConnectStoragePoolVDSCommand(HostName = virt1, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vdsId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:02:49,687-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [4b1c14ef] FINISH, ConnectStoragePoolVDSCommand, log id: 2c999b9b 2017-10-01 16:02:49,687-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler5) [4b1c14ef] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:02:50,809-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [4b1c14ef] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt1 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:02:50,809-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler5) [4b1c14ef] Command 'SpmStatusVDSCommand(HostName = virt1, SpmStatusVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:02:50,809-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler5) [4b1c14ef] Irs placed on server '45f8b331-842e-48e7-9df8-56adddb93836' failed. Proceed Failover 2017-10-01 16:02:50,867-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [4b1c14ef] START, ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 7eeb874e 2017-10-01 16:02:50,867-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [4b1c14ef] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:02:51,521-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [4b1c14ef] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:02:51,521-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [4b1c14ef] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:02:51,521-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [4b1c14ef] HostName = virt2 2017-10-01 16:02:51,521-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [4b1c14ef] Command 'ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:02:51,521-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [4b1c14ef] FINISH, ConnectStoragePoolVDSCommand, log id: 7eeb874e 2017-10-01 16:02:51,521-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler5) [4b1c14ef] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:02:51,710-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [4b1c14ef] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:02:51,710-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler5) [4b1c14ef] Command 'SpmStatusVDSCommand(HostName = virt2, SpmStatusVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:02:51,710-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler5) [4b1c14ef] Irs placed on server 'a0ccfd11-3402-4eba-a119-4987561b738e' failed. Proceed Failover 2017-10-01 16:02:51,767-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [4b1c14ef] START, ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 1545c5d4 2017-10-01 16:02:51,767-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [4b1c14ef] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:02:52,878-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [4b1c14ef] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:02:52,878-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [4b1c14ef] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:02:52,878-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [4b1c14ef] HostName = virt3 2017-10-01 16:02:52,878-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [4b1c14ef] Command 'ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:02:52,878-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [4b1c14ef] FINISH, ConnectStoragePoolVDSCommand, log id: 1545c5d4 2017-10-01 16:02:52,878-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler5) [4b1c14ef] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:02:53,797-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler5) [4b1c14ef] Irs placed on server 'e1e71e9d-a0cd-47db-96c6-f724f6cea736' failed. Proceed Failover 2017-10-01 16:02:53,854-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [4b1c14ef] START, ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 11a92082 2017-10-01 16:02:53,854-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [4b1c14ef] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:02:54,919-04 INFO [org.ovirt.engine.core.bll.storage.domain.ActivateStorageDomainCommand] (default task-216) [78d67821-b42d-49dc-91ec-1a6b2441551c] Lock Acquired to object 'EngineLock:{exclusiveLocks='[c8653dba-63ec-4447-872c-0ffcc6af9c0f=STORAGE]', sharedLocks=''}' 2017-10-01 16:02:54,935-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [4b1c14ef] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:02:54,936-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [4b1c14ef] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:02:54,936-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [4b1c14ef] HostName = virt4 2017-10-01 16:02:54,936-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [4b1c14ef] Command 'ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:02:54,936-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [4b1c14ef] FINISH, ConnectStoragePoolVDSCommand, log id: 11a92082 2017-10-01 16:02:54,936-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler5) [4b1c14ef] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:02:54,996-04 INFO [org.ovirt.engine.core.bll.storage.domain.ActivateStorageDomainCommand] (org.ovirt.thread.pool-6-thread-47) [78d67821-b42d-49dc-91ec-1a6b2441551c] Running command: ActivateStorageDomainCommand internal: false. Entities affected : ID: c8653dba-63ec-4447-872c-0ffcc6af9c0f Type: StorageAction group MANIPULATE_STORAGE_DOMAIN with role type ADMIN 2017-10-01 16:02:55,000-04 INFO [org.ovirt.engine.core.bll.storage.domain.ActivateStorageDomainCommand] (org.ovirt.thread.pool-6-thread-47) [78d67821-b42d-49dc-91ec-1a6b2441551c] Lock freed to object 'EngineLock:{exclusiveLocks='[c8653dba-63ec-4447-872c-0ffcc6af9c0f=STORAGE]', sharedLocks=''}' 2017-10-01 16:02:55,001-04 INFO [org.ovirt.engine.core.bll.storage.domain.ActivateStorageDomainCommand] (org.ovirt.thread.pool-6-thread-47) [78d67821-b42d-49dc-91ec-1a6b2441551c] ActivateStorage Domain. Before Connect all hosts to pool. Time: Sun Oct 01 16:02:55 EDT 2017 2017-10-01 16:02:55,004-04 WARN [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-4) [b4e0c11] Validation of action 'ReconstructMasterDomain' failed for user SYSTEM. Reasons: VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_STORAGE_DOMAIN_STATUS_ILLEGAL2,$status Locked 2017-10-01 16:02:55,007-04 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-6-thread-6) [4b1c14ef] Finished reconstruct for pool 'aca61eac-ed8a-484b-bbee-3645148f6405'. Clearing event queue 2017-10-01 16:02:55,070-04 INFO [org.ovirt.engine.core.bll.storage.connection.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-6-thread-7) [1a9e11e4] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN 2017-10-01 16:02:55,071-04 INFO [org.ovirt.engine.core.bll.storage.connection.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-6-thread-35) [118ae407] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN 2017-10-01 16:02:55,071-04 INFO [org.ovirt.engine.core.bll.storage.connection.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-6-thread-38) [792b4a58] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN 2017-10-01 16:02:55,071-04 INFO [org.ovirt.engine.core.bll.storage.connection.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-6-thread-50) [689b9c73] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN 2017-10-01 16:02:55,086-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-35) [118ae407] START, ConnectStorageServerVDSCommand(HostName = virt4, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='00000000-0000-0000-0000-000000000000', storageType='NFS', connectionList='[StorageServerConnections:{id='5e557659-6817-40fa-9a55-7458125404a6', connection='virtfs-fs:/nfs/data', iqn='null', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 1b1d5b1 2017-10-01 16:02:55,107-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-7) [1a9e11e4] START, ConnectStorageServerVDSCommand(HostName = virt3, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='00000000-0000-0000-0000-000000000000', storageType='NFS', connectionList='[StorageServerConnections:{id='5e557659-6817-40fa-9a55-7458125404a6', connection='virtfs-fs:/nfs/data', iqn='null', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 253bb023 2017-10-01 16:02:55,108-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-38) [792b4a58] START, ConnectStorageServerVDSCommand(HostName = virt1, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='00000000-0000-0000-0000-000000000000', storageType='NFS', connectionList='[StorageServerConnections:{id='5e557659-6817-40fa-9a55-7458125404a6', connection='virtfs-fs:/nfs/data', iqn='null', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 183046c1 2017-10-01 16:02:55,108-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-50) [689b9c73] START, ConnectStorageServerVDSCommand(HostName = virt2, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='00000000-0000-0000-0000-000000000000', storageType='NFS', connectionList='[StorageServerConnections:{id='5e557659-6817-40fa-9a55-7458125404a6', connection='virtfs-fs:/nfs/data', iqn='null', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 1c70abc 2017-10-01 16:02:55,108-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-35) [118ae407] FINISH, ConnectStorageServerVDSCommand, return: {5e557659-6817-40fa-9a55-7458125404a6=0}, log id: 1b1d5b1 2017-10-01 16:02:55,121-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-7) [1a9e11e4] FINISH, ConnectStorageServerVDSCommand, return: {5e557659-6817-40fa-9a55-7458125404a6=0}, log id: 253bb023 2017-10-01 16:02:55,123-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-50) [689b9c73] FINISH, ConnectStorageServerVDSCommand, return: {5e557659-6817-40fa-9a55-7458125404a6=0}, log id: 1c70abc 2017-10-01 16:02:55,131-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-38) [792b4a58] FINISH, ConnectStorageServerVDSCommand, return: {5e557659-6817-40fa-9a55-7458125404a6=0}, log id: 183046c1 2017-10-01 16:02:55,136-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (org.ovirt.thread.pool-6-thread-47) [78d67821-b42d-49dc-91ec-1a6b2441551c] START, ActivateStorageDomainVDSCommand( ActivateStorageDomainVDSCommandParameters:{runAsync='true', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', ignoreFailoverLimit='false', storageDomainId='c8653dba-63ec-4447-872c-0ffcc6af9c0f'}), log id: 7f4efe25 2017-10-01 16:02:55,221-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [78d67821-b42d-49dc-91ec-1a6b2441551c] START, ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 3560981e 2017-10-01 16:02:55,221-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [78d67821-b42d-49dc-91ec-1a6b2441551c] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:02:56,254-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler7) [70e04789] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 16:03:02,115-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-47) [78d67821-b42d-49dc-91ec-1a6b2441551c] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:03:02,115-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [78d67821-b42d-49dc-91ec-1a6b2441551c] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:03:02,115-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [78d67821-b42d-49dc-91ec-1a6b2441551c] HostName = virt3 2017-10-01 16:03:02,115-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [78d67821-b42d-49dc-91ec-1a6b2441551c] Command 'ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:03:02,115-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [78d67821-b42d-49dc-91ec-1a6b2441551c] FINISH, ConnectStoragePoolVDSCommand, log id: 3560981e 2017-10-01 16:03:02,115-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (org.ovirt.thread.pool-6-thread-47) [78d67821-b42d-49dc-91ec-1a6b2441551c] IrsBroker::Failed::ActivateStorageDomainVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:03:02,570-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-47) [78d67821-b42d-49dc-91ec-1a6b2441551c] Irs placed on server 'e1e71e9d-a0cd-47db-96c6-f724f6cea736' failed. Proceed Failover 2017-10-01 16:03:02,633-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [78d67821-b42d-49dc-91ec-1a6b2441551c] START, ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 21308ad2 2017-10-01 16:03:02,633-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [78d67821-b42d-49dc-91ec-1a6b2441551c] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:03:02,904-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-47) [78d67821-b42d-49dc-91ec-1a6b2441551c] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:03:02,904-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [78d67821-b42d-49dc-91ec-1a6b2441551c] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:03:02,904-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [78d67821-b42d-49dc-91ec-1a6b2441551c] HostName = virt2 2017-10-01 16:03:02,904-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [78d67821-b42d-49dc-91ec-1a6b2441551c] Command 'ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:03:02,904-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [78d67821-b42d-49dc-91ec-1a6b2441551c] FINISH, ConnectStoragePoolVDSCommand, log id: 21308ad2 2017-10-01 16:03:02,904-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (org.ovirt.thread.pool-6-thread-47) [78d67821-b42d-49dc-91ec-1a6b2441551c] IrsBroker::Failed::ActivateStorageDomainVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:03:04,030-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-47) [78d67821-b42d-49dc-91ec-1a6b2441551c] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:03:04,030-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (org.ovirt.thread.pool-6-thread-47) [78d67821-b42d-49dc-91ec-1a6b2441551c] Command 'SpmStatusVDSCommand(HostName = virt2, SpmStatusVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:03:04,030-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-47) [78d67821-b42d-49dc-91ec-1a6b2441551c] Irs placed on server 'a0ccfd11-3402-4eba-a119-4987561b738e' failed. Proceed Failover 2017-10-01 16:03:04,095-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [78d67821-b42d-49dc-91ec-1a6b2441551c] START, ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 6a713257 2017-10-01 16:03:04,095-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [78d67821-b42d-49dc-91ec-1a6b2441551c] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:03:04,803-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-47) [78d67821-b42d-49dc-91ec-1a6b2441551c] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:03:04,803-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [78d67821-b42d-49dc-91ec-1a6b2441551c] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:03:04,803-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [78d67821-b42d-49dc-91ec-1a6b2441551c] HostName = virt4 2017-10-01 16:03:04,803-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [78d67821-b42d-49dc-91ec-1a6b2441551c] Command 'ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:03:04,803-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-47) [78d67821-b42d-49dc-91ec-1a6b2441551c] FINISH, ConnectStoragePoolVDSCommand, log id: 6a713257 2017-10-01 16:03:04,803-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (org.ovirt.thread.pool-6-thread-47) [78d67821-b42d-49dc-91ec-1a6b2441551c] IrsBroker::Failed::ActivateStorageDomainVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:03:04,858-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (org.ovirt.thread.pool-6-thread-47) [78d67821-b42d-49dc-91ec-1a6b2441551c] FINISH, ActivateStorageDomainVDSCommand, log id: 7f4efe25 2017-10-01 16:03:04,858-04 ERROR [org.ovirt.engine.core.bll.storage.domain.ActivateStorageDomainCommand] (org.ovirt.thread.pool-6-thread-47) [78d67821-b42d-49dc-91ec-1a6b2441551c] Command 'org.ovirt.engine.core.bll.storage.domain.ActivateStorageDomainCommand' failed: EngineException: org.ovirt.engine.core.vdsbroker.irsbroker.IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' (Failed with error StoragePoolMasterNotFound and code 304) 2017-10-01 16:03:04,861-04 INFO [org.ovirt.engine.core.bll.storage.domain.ActivateStorageDomainCommand] (org.ovirt.thread.pool-6-thread-47) [78d67821-b42d-49dc-91ec-1a6b2441551c] Command [id=bd266e07-9d9f-4b30-ad77-659e2140e02f]: Compensating CHANGED_STATUS_ONLY of org.ovirt.engine.core.common.businessentities.StoragePoolIsoMap; snapshot: EntityStatusSnapshot:{id='StoragePoolIsoMapId:{storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', storageId='c8653dba-63ec-4447-872c-0ffcc6af9c0f'}', status='Inactive'}. 2017-10-01 16:03:04,866-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-47) [78d67821-b42d-49dc-91ec-1a6b2441551c] EVENT_ID: USER_ACTIVATE_STORAGE_DOMAIN_FAILED(967), Correlation ID: 78d67821-b42d-49dc-91ec-1a6b2441551c, Job ID: 3adf65cd-7c3a-4afd-a129-b41c71505cce, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to activate Storage Domain data (Data Center EECS-Primary) by admin@internal 2017-10-01 16:03:06,063-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [69040cca] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:03:06,063-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler1) [69040cca] Command 'SpmStatusVDSCommand(HostName = virt4, SpmStatusVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:03:06,637-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [69040cca] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command HSMGetAllTasksStatusesVDS failed: Not SPM 2017-10-01 16:03:06,637-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler1) [69040cca] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand' return value ' 2017-10-01 16:03:06,637-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler1) [69040cca] HostName = virt4 2017-10-01 16:03:06,637-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler1) [69040cca] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = virt4, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846'})' execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM 2017-10-01 16:03:06,700-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [69040cca] START, ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 2bb0b769 2017-10-01 16:03:06,700-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [69040cca] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:03:06,858-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [69040cca] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:03:06,858-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [69040cca] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:03:06,858-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [69040cca] HostName = virt3 2017-10-01 16:03:06,858-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [69040cca] Command 'ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:03:06,858-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [69040cca] FINISH, ConnectStoragePoolVDSCommand, log id: 2bb0b769 2017-10-01 16:03:06,858-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler1) [69040cca] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:03:07,972-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler1) [69040cca] Irs placed on server 'e1e71e9d-a0cd-47db-96c6-f724f6cea736' failed. Proceed Failover 2017-10-01 16:03:08,030-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [69040cca] START, ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 4a614058 2017-10-01 16:03:08,030-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [69040cca] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:03:08,861-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [69040cca] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:03:08,861-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [69040cca] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:03:08,861-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [69040cca] HostName = virt2 2017-10-01 16:03:08,861-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [69040cca] Command 'ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:03:08,861-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [69040cca] FINISH, ConnectStoragePoolVDSCommand, log id: 4a614058 2017-10-01 16:03:08,861-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler1) [69040cca] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:03:09,983-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [69040cca] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:03:09,983-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler1) [69040cca] Command 'SpmStatusVDSCommand(HostName = virt2, SpmStatusVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:03:09,983-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler1) [69040cca] Irs placed on server 'a0ccfd11-3402-4eba-a119-4987561b738e' failed. Proceed Failover 2017-10-01 16:03:10,041-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [69040cca] START, ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 3785bcae 2017-10-01 16:03:10,041-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [69040cca] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:03:10,314-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [69040cca] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:03:10,314-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [69040cca] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:03:10,314-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [69040cca] HostName = virt4 2017-10-01 16:03:10,314-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [69040cca] Command 'ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:03:10,314-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [69040cca] FINISH, ConnectStoragePoolVDSCommand, log id: 3785bcae 2017-10-01 16:03:10,314-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler1) [69040cca] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:03:11,434-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [69040cca] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:03:11,434-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler1) [69040cca] Command 'SpmStatusVDSCommand(HostName = virt4, SpmStatusVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:03:11,434-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler1) [69040cca] Irs placed on server '5ae29ed2-9c0f-4754-8df3-b43d14726846' failed. Proceed Failover 2017-10-01 16:03:11,492-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [69040cca] START, ConnectStoragePoolVDSCommand(HostName = virt1, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vdsId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 6a70e1f6 2017-10-01 16:03:11,492-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [69040cca] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:03:12,590-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [69040cca] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt1 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:03:12,590-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [69040cca] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:03:12,590-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [69040cca] HostName = virt1 2017-10-01 16:03:12,590-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [69040cca] Command 'ConnectStoragePoolVDSCommand(HostName = virt1, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vdsId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:03:12,590-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [69040cca] FINISH, ConnectStoragePoolVDSCommand, log id: 6a70e1f6 2017-10-01 16:03:12,590-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler1) [69040cca] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:03:12,728-04 INFO [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-44) [120522ad] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: c8653dba-63ec-4447-872c-0ffcc6af9c0f Type: Storage 2017-10-01 16:03:12,747-04 WARN [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-44) [120522ad] skipping connect and refresh for all hosts, last master 'true', command status 'true' 2017-10-01 16:03:12,755-04 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-6-thread-40) [69040cca] Finished reconstruct for pool 'aca61eac-ed8a-484b-bbee-3645148f6405'. Clearing event queue 2017-10-01 16:03:23,719-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [553ade3f] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt1 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:03:23,719-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler3) [553ade3f] Command 'SpmStatusVDSCommand(HostName = virt1, SpmStatusVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:03:23,977-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [553ade3f] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt1 command HSMGetAllTasksStatusesVDS failed: Not SPM 2017-10-01 16:03:23,977-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler3) [553ade3f] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand' return value ' 2017-10-01 16:03:23,977-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler3) [553ade3f] HostName = virt1 2017-10-01 16:03:23,977-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler3) [553ade3f] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = virt1, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836'})' execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM 2017-10-01 16:03:24,039-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [553ade3f] START, ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 6a3d7eae 2017-10-01 16:03:24,039-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [553ade3f] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:03:25,147-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [553ade3f] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:03:25,147-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [553ade3f] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:03:25,147-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [553ade3f] HostName = virt3 2017-10-01 16:03:25,147-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [553ade3f] Command 'ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:03:25,147-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [553ade3f] FINISH, ConnectStoragePoolVDSCommand, log id: 6a3d7eae 2017-10-01 16:03:25,147-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler3) [553ade3f] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:03:25,511-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler3) [553ade3f] Irs placed on server 'e1e71e9d-a0cd-47db-96c6-f724f6cea736' failed. Proceed Failover 2017-10-01 16:03:25,575-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [553ade3f] START, ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 4da18161 2017-10-01 16:03:25,575-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [553ade3f] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:03:26,666-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [553ade3f] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:03:26,666-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [553ade3f] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:03:26,666-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [553ade3f] HostName = virt2 2017-10-01 16:03:26,666-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [553ade3f] Command 'ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:03:26,666-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [553ade3f] FINISH, ConnectStoragePoolVDSCommand, log id: 4da18161 2017-10-01 16:03:26,666-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler3) [553ade3f] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:03:27,786-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [553ade3f] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:03:27,786-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler3) [553ade3f] Command 'SpmStatusVDSCommand(HostName = virt2, SpmStatusVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:03:27,786-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler3) [553ade3f] Irs placed on server 'a0ccfd11-3402-4eba-a119-4987561b738e' failed. Proceed Failover 2017-10-01 16:03:27,846-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [553ade3f] START, ConnectStoragePoolVDSCommand(HostName = virt1, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vdsId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 200e4f8a 2017-10-01 16:03:27,846-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [553ade3f] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:03:28,944-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [553ade3f] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt1 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:03:28,944-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [553ade3f] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:03:28,944-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [553ade3f] HostName = virt1 2017-10-01 16:03:28,944-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [553ade3f] Command 'ConnectStoragePoolVDSCommand(HostName = virt1, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vdsId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:03:28,944-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [553ade3f] FINISH, ConnectStoragePoolVDSCommand, log id: 200e4f8a 2017-10-01 16:03:28,944-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler3) [553ade3f] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:03:30,065-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [553ade3f] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt1 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:03:30,066-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler3) [553ade3f] Command 'SpmStatusVDSCommand(HostName = virt1, SpmStatusVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:03:30,066-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler3) [553ade3f] Irs placed on server '45f8b331-842e-48e7-9df8-56adddb93836' failed. Proceed Failover 2017-10-01 16:03:30,123-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [553ade3f] START, ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 4f69e428 2017-10-01 16:03:30,123-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [553ade3f] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:03:31,202-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [553ade3f] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:03:31,202-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [553ade3f] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:03:31,202-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [553ade3f] HostName = virt4 2017-10-01 16:03:31,202-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [553ade3f] Command 'ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:03:31,202-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [553ade3f] FINISH, ConnectStoragePoolVDSCommand, log id: 4f69e428 2017-10-01 16:03:31,202-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler3) [553ade3f] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:03:31,339-04 INFO [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-19) [115465cd] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: c8653dba-63ec-4447-872c-0ffcc6af9c0f Type: Storage 2017-10-01 16:03:31,347-04 WARN [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-19) [115465cd] skipping connect and refresh for all hosts, last master 'true', command status 'true' 2017-10-01 16:03:31,357-04 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-6-thread-13) [553ade3f] Finished reconstruct for pool 'aca61eac-ed8a-484b-bbee-3645148f6405'. Clearing event queue 2017-10-01 16:03:42,330-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [585bc766] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:03:42,330-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler10) [585bc766] Command 'SpmStatusVDSCommand(HostName = virt4, SpmStatusVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:03:43,339-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [585bc766] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command HSMGetAllTasksStatusesVDS failed: Not SPM 2017-10-01 16:03:43,339-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler10) [585bc766] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand' return value ' 2017-10-01 16:03:43,339-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler10) [585bc766] HostName = virt4 2017-10-01 16:03:43,339-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler10) [585bc766] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = virt4, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846'})' execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM 2017-10-01 16:03:43,407-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [585bc766] START, ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 39ab7dbe 2017-10-01 16:03:43,407-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [585bc766] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:03:44,482-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [585bc766] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:03:44,482-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [585bc766] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:03:44,482-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [585bc766] HostName = virt4 2017-10-01 16:03:44,482-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [585bc766] Command 'ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:03:44,482-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [585bc766] FINISH, ConnectStoragePoolVDSCommand, log id: 39ab7dbe 2017-10-01 16:03:44,482-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler10) [585bc766] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:03:45,608-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [585bc766] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:03:45,608-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler10) [585bc766] Command 'SpmStatusVDSCommand(HostName = virt4, SpmStatusVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:03:45,608-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler10) [585bc766] Irs placed on server '5ae29ed2-9c0f-4754-8df3-b43d14726846' failed. Proceed Failover 2017-10-01 16:03:45,665-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [585bc766] START, ConnectStoragePoolVDSCommand(HostName = virt1, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vdsId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 2f001314 2017-10-01 16:03:45,665-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [585bc766] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:03:48,214-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [585bc766] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt1 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:03:48,214-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [585bc766] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:03:48,214-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [585bc766] HostName = virt1 2017-10-01 16:03:48,214-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [585bc766] Command 'ConnectStoragePoolVDSCommand(HostName = virt1, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vdsId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:03:48,214-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [585bc766] FINISH, ConnectStoragePoolVDSCommand, log id: 2f001314 2017-10-01 16:03:48,214-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler10) [585bc766] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:03:49,118-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [585bc766] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt1 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:03:49,118-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler10) [585bc766] Command 'SpmStatusVDSCommand(HostName = virt1, SpmStatusVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:03:49,118-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler10) [585bc766] Irs placed on server '45f8b331-842e-48e7-9df8-56adddb93836' failed. Proceed Failover 2017-10-01 16:03:49,177-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [585bc766] START, ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 5c903cdb 2017-10-01 16:03:49,177-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [585bc766] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:03:50,267-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [585bc766] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:03:50,267-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [585bc766] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:03:50,267-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [585bc766] HostName = virt2 2017-10-01 16:03:50,267-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [585bc766] Command 'ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:03:50,267-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [585bc766] FINISH, ConnectStoragePoolVDSCommand, log id: 5c903cdb 2017-10-01 16:03:50,267-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler10) [585bc766] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:03:51,388-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [585bc766] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:03:51,388-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler10) [585bc766] Command 'SpmStatusVDSCommand(HostName = virt2, SpmStatusVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:03:51,388-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler10) [585bc766] Irs placed on server 'a0ccfd11-3402-4eba-a119-4987561b738e' failed. Proceed Failover 2017-10-01 16:03:51,445-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [585bc766] START, ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 154b6d97 2017-10-01 16:03:51,445-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [585bc766] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:03:52,069-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [585bc766] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:03:52,069-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [585bc766] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:03:52,069-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [585bc766] HostName = virt3 2017-10-01 16:03:52,069-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [585bc766] Command 'ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:03:52,069-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [585bc766] FINISH, ConnectStoragePoolVDSCommand, log id: 154b6d97 2017-10-01 16:03:52,069-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler10) [585bc766] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:03:52,208-04 INFO [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-12) [555c0de4] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: c8653dba-63ec-4447-872c-0ffcc6af9c0f Type: Storage 2017-10-01 16:03:52,221-04 WARN [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-12) [555c0de4] skipping connect and refresh for all hosts, last master 'true', command status 'true' 2017-10-01 16:03:52,232-04 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-6-thread-42) [585bc766] Finished reconstruct for pool 'aca61eac-ed8a-484b-bbee-3645148f6405'. Clearing event queue 2017-10-01 16:03:56,272-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler7) [cbdb3ae] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 16:04:03,415-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [2baa1da9] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command HSMGetAllTasksStatusesVDS failed: Not SPM: () 2017-10-01 16:04:03,415-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler3) [2baa1da9] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand' return value ' 2017-10-01 16:04:03,415-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler3) [2baa1da9] HostName = virt3 2017-10-01 16:04:03,415-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler3) [2baa1da9] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = virt3, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736'})' execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () 2017-10-01 16:04:03,477-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [2baa1da9] START, ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 643cb696 2017-10-01 16:04:03,477-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [2baa1da9] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:04:04,183-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [2baa1da9] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:04:04,183-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [2baa1da9] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:04:04,183-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [2baa1da9] HostName = virt4 2017-10-01 16:04:04,183-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [2baa1da9] Command 'ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:04:04,183-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [2baa1da9] FINISH, ConnectStoragePoolVDSCommand, log id: 643cb696 2017-10-01 16:04:04,183-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler3) [2baa1da9] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:04:05,308-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [2baa1da9] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:04:05,308-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler3) [2baa1da9] Command 'SpmStatusVDSCommand(HostName = virt4, SpmStatusVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:04:05,308-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler3) [2baa1da9] Irs placed on server '5ae29ed2-9c0f-4754-8df3-b43d14726846' failed. Proceed Failover 2017-10-01 16:04:05,367-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [2baa1da9] START, ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 64d8f639 2017-10-01 16:04:05,367-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [2baa1da9] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:04:07,483-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [2baa1da9] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:04:07,483-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [2baa1da9] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:04:07,483-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [2baa1da9] HostName = virt3 2017-10-01 16:04:07,483-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [2baa1da9] Command 'ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:04:07,483-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [2baa1da9] FINISH, ConnectStoragePoolVDSCommand, log id: 64d8f639 2017-10-01 16:04:07,483-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler3) [2baa1da9] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:04:07,800-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler3) [2baa1da9] Irs placed on server 'e1e71e9d-a0cd-47db-96c6-f724f6cea736' failed. Proceed Failover 2017-10-01 16:04:07,858-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [2baa1da9] START, ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 318a1e00 2017-10-01 16:04:07,859-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [2baa1da9] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:04:08,809-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [2baa1da9] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:04:08,809-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [2baa1da9] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:04:08,809-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [2baa1da9] HostName = virt2 2017-10-01 16:04:08,809-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [2baa1da9] Command 'ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:04:08,809-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [2baa1da9] FINISH, ConnectStoragePoolVDSCommand, log id: 318a1e00 2017-10-01 16:04:08,809-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler3) [2baa1da9] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:04:09,929-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [2baa1da9] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:04:09,929-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler3) [2baa1da9] Command 'SpmStatusVDSCommand(HostName = virt2, SpmStatusVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:04:09,929-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler3) [2baa1da9] Irs placed on server 'a0ccfd11-3402-4eba-a119-4987561b738e' failed. Proceed Failover 2017-10-01 16:04:09,987-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [2baa1da9] START, ConnectStoragePoolVDSCommand(HostName = virt1, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vdsId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 5931d0fa 2017-10-01 16:04:09,987-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [2baa1da9] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:04:11,055-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [2baa1da9] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt1 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:04:11,055-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [2baa1da9] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:04:11,055-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [2baa1da9] HostName = virt1 2017-10-01 16:04:11,055-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [2baa1da9] Command 'ConnectStoragePoolVDSCommand(HostName = virt1, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vdsId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:04:11,055-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [2baa1da9] FINISH, ConnectStoragePoolVDSCommand, log id: 5931d0fa 2017-10-01 16:04:11,055-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler3) [2baa1da9] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:04:11,181-04 INFO [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-50) [18633d7b] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: c8653dba-63ec-4447-872c-0ffcc6af9c0f Type: Storage 2017-10-01 16:04:11,191-04 WARN [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-50) [18633d7b] skipping connect and refresh for all hosts, last master 'true', command status 'true' 2017-10-01 16:04:11,198-04 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-6-thread-7) [2baa1da9] Finished reconstruct for pool 'aca61eac-ed8a-484b-bbee-3645148f6405'. Clearing event queue 2017-10-01 16:04:22,178-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [7a720243] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt1 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:04:22,178-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler7) [7a720243] Command 'SpmStatusVDSCommand(HostName = virt1, SpmStatusVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:04:22,866-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [7a720243] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt1 command HSMGetAllTasksStatusesVDS failed: Not SPM 2017-10-01 16:04:22,866-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler7) [7a720243] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand' return value ' 2017-10-01 16:04:22,866-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler7) [7a720243] HostName = virt1 2017-10-01 16:04:22,866-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler7) [7a720243] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = virt1, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836'})' execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM 2017-10-01 16:04:22,928-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [7a720243] START, ConnectStoragePoolVDSCommand(HostName = virt1, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vdsId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 68afd887 2017-10-01 16:04:22,928-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [7a720243] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:04:23,927-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [7a720243] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt1 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:04:23,927-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [7a720243] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:04:23,927-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [7a720243] HostName = virt1 2017-10-01 16:04:23,927-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [7a720243] Command 'ConnectStoragePoolVDSCommand(HostName = virt1, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vdsId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:04:23,927-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [7a720243] FINISH, ConnectStoragePoolVDSCommand, log id: 68afd887 2017-10-01 16:04:23,927-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler7) [7a720243] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:04:25,049-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [7a720243] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt1 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:04:25,049-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler7) [7a720243] Command 'SpmStatusVDSCommand(HostName = virt1, SpmStatusVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:04:25,049-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler7) [7a720243] Irs placed on server '45f8b331-842e-48e7-9df8-56adddb93836' failed. Proceed Failover 2017-10-01 16:04:25,107-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [7a720243] START, ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 43b7c37b 2017-10-01 16:04:25,107-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [7a720243] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:04:27,059-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [7a720243] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:04:27,059-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [7a720243] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:04:27,059-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [7a720243] HostName = virt4 2017-10-01 16:04:27,059-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [7a720243] Command 'ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:04:27,059-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [7a720243] FINISH, ConnectStoragePoolVDSCommand, log id: 43b7c37b 2017-10-01 16:04:27,059-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler7) [7a720243] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:04:28,177-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [7a720243] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:04:28,177-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler7) [7a720243] Command 'SpmStatusVDSCommand(HostName = virt4, SpmStatusVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:04:28,177-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler7) [7a720243] Irs placed on server '5ae29ed2-9c0f-4754-8df3-b43d14726846' failed. Proceed Failover 2017-10-01 16:04:28,235-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [7a720243] START, ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 75a62085 2017-10-01 16:04:28,235-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [7a720243] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:04:34,138-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [7a720243] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:04:34,138-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [7a720243] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:04:34,138-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [7a720243] HostName = virt3 2017-10-01 16:04:34,138-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [7a720243] Command 'ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:04:34,138-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [7a720243] FINISH, ConnectStoragePoolVDSCommand, log id: 75a62085 2017-10-01 16:04:34,138-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler7) [7a720243] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:04:34,397-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler7) [7a720243] Irs placed on server 'e1e71e9d-a0cd-47db-96c6-f724f6cea736' failed. Proceed Failover 2017-10-01 16:04:34,466-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [7a720243] START, ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 29e1d28b 2017-10-01 16:04:34,466-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [7a720243] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:04:35,308-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [7a720243] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:04:35,308-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [7a720243] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:04:35,308-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [7a720243] HostName = virt2 2017-10-01 16:04:35,308-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [7a720243] Command 'ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:04:35,308-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [7a720243] FINISH, ConnectStoragePoolVDSCommand, log id: 29e1d28b 2017-10-01 16:04:35,308-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler7) [7a720243] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:04:35,444-04 INFO [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-40) [2743b82] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: c8653dba-63ec-4447-872c-0ffcc6af9c0f Type: Storage 2017-10-01 16:04:35,457-04 WARN [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-40) [2743b82] skipping connect and refresh for all hosts, last master 'true', command status 'true' 2017-10-01 16:04:35,466-04 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-6-thread-44) [7a720243] Finished reconstruct for pool 'aca61eac-ed8a-484b-bbee-3645148f6405'. Clearing event queue 2017-10-01 16:04:46,430-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [1e85e42] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:04:46,430-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler7) [1e85e42] Command 'SpmStatusVDSCommand(HostName = virt2, SpmStatusVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:04:47,439-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [1e85e42] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command HSMGetAllTasksStatusesVDS failed: Not SPM 2017-10-01 16:04:47,440-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler7) [1e85e42] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand' return value ' 2017-10-01 16:04:47,440-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler7) [1e85e42] HostName = virt2 2017-10-01 16:04:47,440-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler7) [1e85e42] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = virt2, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e'})' execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM 2017-10-01 16:04:47,502-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [1e85e42] START, ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 2d808f77 2017-10-01 16:04:47,502-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [1e85e42] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:04:48,582-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [1e85e42] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:04:48,582-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [1e85e42] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:04:48,582-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [1e85e42] HostName = virt4 2017-10-01 16:04:48,582-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [1e85e42] Command 'ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:04:48,582-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [1e85e42] FINISH, ConnectStoragePoolVDSCommand, log id: 2d808f77 2017-10-01 16:04:48,582-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler7) [1e85e42] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:04:49,562-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [1e85e42] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:04:49,562-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler7) [1e85e42] Command 'SpmStatusVDSCommand(HostName = virt4, SpmStatusVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:04:49,562-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler7) [1e85e42] Irs placed on server '5ae29ed2-9c0f-4754-8df3-b43d14726846' failed. Proceed Failover 2017-10-01 16:04:49,620-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [1e85e42] START, ConnectStoragePoolVDSCommand(HostName = virt1, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vdsId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 3c31aebb 2017-10-01 16:04:49,620-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [1e85e42] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:04:50,717-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [1e85e42] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt1 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:04:50,717-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [1e85e42] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:04:50,718-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [1e85e42] HostName = virt1 2017-10-01 16:04:50,718-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [1e85e42] Command 'ConnectStoragePoolVDSCommand(HostName = virt1, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vdsId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:04:50,718-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [1e85e42] FINISH, ConnectStoragePoolVDSCommand, log id: 3c31aebb 2017-10-01 16:04:50,718-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler7) [1e85e42] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:04:51,283-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [1e85e42] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt1 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:04:51,283-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler7) [1e85e42] Command 'SpmStatusVDSCommand(HostName = virt1, SpmStatusVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:04:51,283-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler7) [1e85e42] Irs placed on server '45f8b331-842e-48e7-9df8-56adddb93836' failed. Proceed Failover 2017-10-01 16:04:51,341-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [1e85e42] START, ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 5270a07d 2017-10-01 16:04:51,341-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [1e85e42] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:04:52,378-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [1e85e42] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:04:52,378-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [1e85e42] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:04:52,378-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [1e85e42] HostName = virt2 2017-10-01 16:04:52,378-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [1e85e42] Command 'ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:04:52,378-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [1e85e42] FINISH, ConnectStoragePoolVDSCommand, log id: 5270a07d 2017-10-01 16:04:52,378-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler7) [1e85e42] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:04:53,501-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [1e85e42] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:04:53,501-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler7) [1e85e42] Command 'SpmStatusVDSCommand(HostName = virt2, SpmStatusVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:04:53,501-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler7) [1e85e42] Irs placed on server 'a0ccfd11-3402-4eba-a119-4987561b738e' failed. Proceed Failover 2017-10-01 16:04:53,560-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [1e85e42] START, ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 130315eb 2017-10-01 16:04:53,560-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [1e85e42] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:04:53,939-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [1e85e42] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:04:53,939-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [1e85e42] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:04:53,939-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [1e85e42] HostName = virt3 2017-10-01 16:04:53,939-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [1e85e42] Command 'ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:04:53,939-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [1e85e42] FINISH, ConnectStoragePoolVDSCommand, log id: 130315eb 2017-10-01 16:04:53,939-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler7) [1e85e42] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:04:54,070-04 INFO [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-11) [3175a93c] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: c8653dba-63ec-4447-872c-0ffcc6af9c0f Type: Storage 2017-10-01 16:04:54,082-04 WARN [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-11) [3175a93c] skipping connect and refresh for all hosts, last master 'true', command status 'true' 2017-10-01 16:04:54,089-04 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-6-thread-10) [1e85e42] Finished reconstruct for pool 'aca61eac-ed8a-484b-bbee-3645148f6405'. Clearing event queue 2017-10-01 16:04:56,291-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [641af659] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 16:05:00,015-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [15b4e9d4] Autorecovering 1 storage domains 2017-10-01 16:05:00,015-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [15b4e9d4] Autorecovering storage domains id: c8653dba-63ec-4447-872c-0ffcc6af9c0f 2017-10-01 16:05:00,019-04 INFO [org.ovirt.engine.core.bll.storage.connection.ConnectDomainToStorageCommand] (DefaultQuartzScheduler6) [916bcf3] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: c8653dba-63ec-4447-872c-0ffcc6af9c0f Type: Storage 2017-10-01 16:05:00,019-04 INFO [org.ovirt.engine.core.bll.storage.connection.ConnectDomainToStorageCommand] (DefaultQuartzScheduler6) [916bcf3] ConnectDomainToStorage. Before Connect all hosts to pool. Time: Sun Oct 01 16:05:00 EDT 2017 2017-10-01 16:05:00,081-04 INFO [org.ovirt.engine.core.bll.storage.connection.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-6-thread-1) [2e1f9b03] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN 2017-10-01 16:05:00,081-04 INFO [org.ovirt.engine.core.bll.storage.connection.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-6-thread-15) [4fd06568] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN 2017-10-01 16:05:00,085-04 INFO [org.ovirt.engine.core.bll.storage.connection.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-6-thread-23) [325c5505] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN 2017-10-01 16:05:00,085-04 INFO [org.ovirt.engine.core.bll.storage.connection.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-6-thread-20) [2ffe8dde] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN 2017-10-01 16:05:00,096-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-15) [4fd06568] START, ConnectStorageServerVDSCommand(HostName = virt3, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='00000000-0000-0000-0000-000000000000', storageType='NFS', connectionList='[StorageServerConnections:{id='5e557659-6817-40fa-9a55-7458125404a6', connection='virtfs-fs:/nfs/data', iqn='null', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 5d2ab28e 2017-10-01 16:05:00,100-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-23) [325c5505] START, ConnectStorageServerVDSCommand(HostName = virt2, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='00000000-0000-0000-0000-000000000000', storageType='NFS', connectionList='[StorageServerConnections:{id='5e557659-6817-40fa-9a55-7458125404a6', connection='virtfs-fs:/nfs/data', iqn='null', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 6d2e8820 2017-10-01 16:05:00,100-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-20) [2ffe8dde] START, ConnectStorageServerVDSCommand(HostName = virt4, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='00000000-0000-0000-0000-000000000000', storageType='NFS', connectionList='[StorageServerConnections:{id='5e557659-6817-40fa-9a55-7458125404a6', connection='virtfs-fs:/nfs/data', iqn='null', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 5c268698 2017-10-01 16:05:00,111-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-15) [4fd06568] FINISH, ConnectStorageServerVDSCommand, return: {5e557659-6817-40fa-9a55-7458125404a6=0}, log id: 5d2ab28e 2017-10-01 16:05:00,112-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-20) [2ffe8dde] FINISH, ConnectStorageServerVDSCommand, return: {5e557659-6817-40fa-9a55-7458125404a6=0}, log id: 5c268698 2017-10-01 16:05:00,117-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-1) [2e1f9b03] START, ConnectStorageServerVDSCommand(HostName = virt1, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='00000000-0000-0000-0000-000000000000', storageType='NFS', connectionList='[StorageServerConnections:{id='5e557659-6817-40fa-9a55-7458125404a6', connection='virtfs-fs:/nfs/data', iqn='null', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 101ef3b9 2017-10-01 16:05:00,118-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-23) [325c5505] FINISH, ConnectStorageServerVDSCommand, return: {5e557659-6817-40fa-9a55-7458125404a6=0}, log id: 6d2e8820 2017-10-01 16:05:00,506-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-1) [2e1f9b03] FINISH, ConnectStorageServerVDSCommand, return: {5e557659-6817-40fa-9a55-7458125404a6=0}, log id: 101ef3b9 2017-10-01 16:05:00,509-04 INFO [org.ovirt.engine.core.bll.storage.connection.ConnectDomainToStorageCommand] (DefaultQuartzScheduler6) [916bcf3] ConnectDomainToStorage. After Connect all hosts to pool. Time: Sun Oct 01 16:05:00 EDT 2017 2017-10-01 16:05:04,722-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [285e039a] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command HSMGetAllTasksStatusesVDS failed: Not SPM: () 2017-10-01 16:05:04,722-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler7) [285e039a] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand' return value ' 2017-10-01 16:05:04,722-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler7) [285e039a] HostName = virt3 2017-10-01 16:05:04,722-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler7) [285e039a] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = virt3, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736'})' execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () 2017-10-01 16:05:04,784-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [285e039a] START, ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 15bc43a4 2017-10-01 16:05:04,784-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [285e039a] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:05:05,873-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [285e039a] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:05:05,873-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [285e039a] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:05:05,873-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [285e039a] HostName = virt2 2017-10-01 16:05:05,874-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [285e039a] Command 'ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:05:05,874-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [285e039a] FINISH, ConnectStoragePoolVDSCommand, log id: 15bc43a4 2017-10-01 16:05:05,874-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler7) [285e039a] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:05:06,350-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [285e039a] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:05:06,350-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler7) [285e039a] Command 'SpmStatusVDSCommand(HostName = virt2, SpmStatusVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:05:06,350-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler7) [285e039a] Irs placed on server 'a0ccfd11-3402-4eba-a119-4987561b738e' failed. Proceed Failover 2017-10-01 16:05:06,407-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [285e039a] START, ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 47851097 2017-10-01 16:05:06,407-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [285e039a] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:05:07,482-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [285e039a] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:05:07,482-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [285e039a] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:05:07,482-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [285e039a] HostName = virt4 2017-10-01 16:05:07,482-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [285e039a] Command 'ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:05:07,482-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [285e039a] FINISH, ConnectStoragePoolVDSCommand, log id: 47851097 2017-10-01 16:05:07,482-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler7) [285e039a] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:05:07,602-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [285e039a] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:05:07,602-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler7) [285e039a] Command 'SpmStatusVDSCommand(HostName = virt4, SpmStatusVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:05:07,602-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler7) [285e039a] Irs placed on server '5ae29ed2-9c0f-4754-8df3-b43d14726846' failed. Proceed Failover 2017-10-01 16:05:07,660-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [285e039a] START, ConnectStoragePoolVDSCommand(HostName = virt1, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vdsId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: d0613bf 2017-10-01 16:05:07,660-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [285e039a] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:05:08,765-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [285e039a] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt1 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:05:08,765-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [285e039a] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:05:08,765-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [285e039a] HostName = virt1 2017-10-01 16:05:08,765-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [285e039a] Command 'ConnectStoragePoolVDSCommand(HostName = virt1, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vdsId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:05:08,765-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [285e039a] FINISH, ConnectStoragePoolVDSCommand, log id: d0613bf 2017-10-01 16:05:08,765-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler7) [285e039a] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:05:09,891-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [285e039a] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt1 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:05:09,891-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler7) [285e039a] Command 'SpmStatusVDSCommand(HostName = virt1, SpmStatusVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:05:09,891-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler7) [285e039a] Irs placed on server '45f8b331-842e-48e7-9df8-56adddb93836' failed. Proceed Failover 2017-10-01 16:05:09,955-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [285e039a] START, ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 55137f28 2017-10-01 16:05:09,955-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [285e039a] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:05:11,056-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [285e039a] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:05:11,056-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [285e039a] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:05:11,056-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [285e039a] HostName = virt3 2017-10-01 16:05:11,056-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [285e039a] Command 'ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:05:11,056-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [285e039a] FINISH, ConnectStoragePoolVDSCommand, log id: 55137f28 2017-10-01 16:05:11,056-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler7) [285e039a] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:05:11,190-04 INFO [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-17) [44ffd746] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: c8653dba-63ec-4447-872c-0ffcc6af9c0f Type: Storage 2017-10-01 16:05:11,254-04 WARN [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-17) [44ffd746] skipping connect and refresh for all hosts, last master 'true', command status 'true' 2017-10-01 16:05:11,264-04 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-6-thread-30) [285e039a] Finished reconstruct for pool 'aca61eac-ed8a-484b-bbee-3645148f6405'. Clearing event queue 2017-10-01 16:05:22,419-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [7d8b7023] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command HSMGetAllTasksStatusesVDS failed: Not SPM: () 2017-10-01 16:05:22,419-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler8) [7d8b7023] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand' return value ' 2017-10-01 16:05:22,419-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler8) [7d8b7023] HostName = virt3 2017-10-01 16:05:22,419-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler8) [7d8b7023] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = virt3, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736'})' execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () 2017-10-01 16:05:22,481-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [7d8b7023] START, ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 58d8f52e 2017-10-01 16:05:22,481-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [7d8b7023] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:05:22,754-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [7d8b7023] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:05:22,754-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [7d8b7023] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:05:22,754-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [7d8b7023] HostName = virt2 2017-10-01 16:05:22,754-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [7d8b7023] Command 'ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:05:22,754-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [7d8b7023] FINISH, ConnectStoragePoolVDSCommand, log id: 58d8f52e 2017-10-01 16:05:22,754-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler8) [7d8b7023] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:05:23,873-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [7d8b7023] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:05:23,873-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler8) [7d8b7023] Command 'SpmStatusVDSCommand(HostName = virt2, SpmStatusVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:05:23,873-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler8) [7d8b7023] Irs placed on server 'a0ccfd11-3402-4eba-a119-4987561b738e' failed. Proceed Failover 2017-10-01 16:05:23,933-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [7d8b7023] START, ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 94c3db6 2017-10-01 16:05:23,933-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [7d8b7023] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:05:24,962-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [7d8b7023] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:05:24,962-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [7d8b7023] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:05:24,962-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [7d8b7023] HostName = virt3 2017-10-01 16:05:24,962-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [7d8b7023] Command 'ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:05:24,962-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [7d8b7023] FINISH, ConnectStoragePoolVDSCommand, log id: 94c3db6 2017-10-01 16:05:24,962-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler8) [7d8b7023] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:05:26,075-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler8) [7d8b7023] Irs placed on server 'e1e71e9d-a0cd-47db-96c6-f724f6cea736' failed. Proceed Failover 2017-10-01 16:05:26,133-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [7d8b7023] START, ConnectStoragePoolVDSCommand(HostName = virt1, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vdsId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 607400d6 2017-10-01 16:05:26,133-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [7d8b7023] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:05:27,062-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [7d8b7023] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt1 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:05:27,062-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [7d8b7023] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:05:27,062-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [7d8b7023] HostName = virt1 2017-10-01 16:05:27,062-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [7d8b7023] Command 'ConnectStoragePoolVDSCommand(HostName = virt1, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vdsId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:05:27,062-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [7d8b7023] FINISH, ConnectStoragePoolVDSCommand, log id: 607400d6 2017-10-01 16:05:27,062-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler8) [7d8b7023] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:05:28,182-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [7d8b7023] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt1 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:05:28,182-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler8) [7d8b7023] Command 'SpmStatusVDSCommand(HostName = virt1, SpmStatusVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:05:28,182-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler8) [7d8b7023] Irs placed on server '45f8b331-842e-48e7-9df8-56adddb93836' failed. Proceed Failover 2017-10-01 16:05:28,239-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [7d8b7023] START, ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 1907a5fd 2017-10-01 16:05:28,239-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [7d8b7023] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:05:28,596-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [7d8b7023] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:05:28,596-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [7d8b7023] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:05:28,596-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [7d8b7023] HostName = virt4 2017-10-01 16:05:28,596-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [7d8b7023] Command 'ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:05:28,596-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [7d8b7023] FINISH, ConnectStoragePoolVDSCommand, log id: 1907a5fd 2017-10-01 16:05:28,596-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler8) [7d8b7023] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:05:28,724-04 INFO [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-7) [646142fd] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: c8653dba-63ec-4447-872c-0ffcc6af9c0f Type: Storage 2017-10-01 16:05:28,732-04 WARN [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-7) [646142fd] skipping connect and refresh for all hosts, last master 'true', command status 'true' 2017-10-01 16:05:28,739-04 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-6-thread-50) [7d8b7023] Finished reconstruct for pool 'aca61eac-ed8a-484b-bbee-3645148f6405'. Clearing event queue 2017-10-01 16:05:39,723-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [5b428178] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:05:39,723-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler10) [5b428178] Command 'SpmStatusVDSCommand(HostName = virt4, SpmStatusVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:05:40,077-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [5b428178] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command HSMGetAllTasksStatusesVDS failed: Not SPM 2017-10-01 16:05:40,077-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler10) [5b428178] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand' return value ' 2017-10-01 16:05:40,077-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler10) [5b428178] HostName = virt4 2017-10-01 16:05:40,077-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler10) [5b428178] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = virt4, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846'})' execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM 2017-10-01 16:05:40,145-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [5b428178] START, ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 1cc2b1ea 2017-10-01 16:05:40,145-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [5b428178] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:05:41,223-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [5b428178] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:05:41,223-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [5b428178] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:05:41,223-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [5b428178] HostName = virt4 2017-10-01 16:05:41,223-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [5b428178] Command 'ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:05:41,223-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [5b428178] FINISH, ConnectStoragePoolVDSCommand, log id: 1cc2b1ea 2017-10-01 16:05:41,223-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler10) [5b428178] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:05:42,179-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [5b428178] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:05:42,179-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler10) [5b428178] Command 'SpmStatusVDSCommand(HostName = virt4, SpmStatusVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:05:42,179-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler10) [5b428178] Irs placed on server '5ae29ed2-9c0f-4754-8df3-b43d14726846' failed. Proceed Failover 2017-10-01 16:05:42,237-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [5b428178] START, ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 45b16c4a 2017-10-01 16:05:42,237-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [5b428178] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:05:43,345-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [5b428178] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:05:43,345-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [5b428178] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:05:43,345-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [5b428178] HostName = virt3 2017-10-01 16:05:43,345-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [5b428178] Command 'ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:05:43,345-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [5b428178] FINISH, ConnectStoragePoolVDSCommand, log id: 45b16c4a 2017-10-01 16:05:43,345-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler10) [5b428178] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:05:44,458-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler10) [5b428178] Irs placed on server 'e1e71e9d-a0cd-47db-96c6-f724f6cea736' failed. Proceed Failover 2017-10-01 16:05:44,522-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [5b428178] START, ConnectStoragePoolVDSCommand(HostName = virt1, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vdsId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 275ab07b 2017-10-01 16:05:44,522-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [5b428178] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:05:44,742-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [5b428178] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt1 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:05:44,742-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [5b428178] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:05:44,742-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [5b428178] HostName = virt1 2017-10-01 16:05:44,742-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [5b428178] Command 'ConnectStoragePoolVDSCommand(HostName = virt1, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vdsId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:05:44,742-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [5b428178] FINISH, ConnectStoragePoolVDSCommand, log id: 275ab07b 2017-10-01 16:05:44,742-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler10) [5b428178] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:05:45,862-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [5b428178] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt1 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:05:45,862-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler10) [5b428178] Command 'SpmStatusVDSCommand(HostName = virt1, SpmStatusVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:05:45,862-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler10) [5b428178] Irs placed on server '45f8b331-842e-48e7-9df8-56adddb93836' failed. Proceed Failover 2017-10-01 16:05:45,919-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [5b428178] START, ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 4cbf47b4 2017-10-01 16:05:45,920-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [5b428178] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:05:46,194-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [5b428178] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:05:46,194-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [5b428178] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:05:46,194-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [5b428178] HostName = virt2 2017-10-01 16:05:46,194-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [5b428178] Command 'ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:05:46,194-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [5b428178] FINISH, ConnectStoragePoolVDSCommand, log id: 4cbf47b4 2017-10-01 16:05:46,194-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler10) [5b428178] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:05:46,321-04 INFO [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-41) [52d53de3] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: c8653dba-63ec-4447-872c-0ffcc6af9c0f Type: Storage 2017-10-01 16:05:46,332-04 WARN [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-41) [52d53de3] skipping connect and refresh for all hosts, last master 'true', command status 'true' 2017-10-01 16:05:46,341-04 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-6-thread-49) [5b428178] Finished reconstruct for pool 'aca61eac-ed8a-484b-bbee-3645148f6405'. Clearing event queue 2017-10-01 16:05:56,314-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [3ef839ea] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 16:05:57,244-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [2e8f57f9] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:05:57,244-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler2) [2e8f57f9] Command 'SpmStatusVDSCommand(HostName = virt2, SpmStatusVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:05:58,253-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [2e8f57f9] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command HSMGetAllTasksStatusesVDS failed: Not SPM 2017-10-01 16:05:58,254-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler2) [2e8f57f9] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand' return value ' 2017-10-01 16:05:58,254-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler2) [2e8f57f9] HostName = virt2 2017-10-01 16:05:58,254-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler2) [2e8f57f9] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = virt2, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e'})' execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM 2017-10-01 16:05:58,315-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [2e8f57f9] START, ConnectStoragePoolVDSCommand(HostName = virt1, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vdsId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 32efdcda 2017-10-01 16:05:58,315-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [2e8f57f9] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:05:59,419-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [2e8f57f9] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt1 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:05:59,419-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [2e8f57f9] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:05:59,419-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [2e8f57f9] HostName = virt1 2017-10-01 16:05:59,419-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [2e8f57f9] Command 'ConnectStoragePoolVDSCommand(HostName = virt1, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vdsId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:05:59,419-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [2e8f57f9] FINISH, ConnectStoragePoolVDSCommand, log id: 32efdcda 2017-10-01 16:05:59,419-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler2) [2e8f57f9] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:05:59,943-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [2e8f57f9] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt1 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:05:59,943-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler2) [2e8f57f9] Command 'SpmStatusVDSCommand(HostName = virt1, SpmStatusVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:05:59,943-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler2) [2e8f57f9] Irs placed on server '45f8b331-842e-48e7-9df8-56adddb93836' failed. Proceed Failover 2017-10-01 16:06:00,000-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [2e8f57f9] START, ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 19d158c0 2017-10-01 16:06:00,000-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [2e8f57f9] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:06:01,090-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [2e8f57f9] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:06:01,090-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [2e8f57f9] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:06:01,090-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [2e8f57f9] HostName = virt2 2017-10-01 16:06:01,090-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [2e8f57f9] Command 'ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:06:01,090-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [2e8f57f9] FINISH, ConnectStoragePoolVDSCommand, log id: 19d158c0 2017-10-01 16:06:01,090-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler2) [2e8f57f9] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:06:02,198-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [2e8f57f9] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:06:02,198-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler2) [2e8f57f9] Command 'SpmStatusVDSCommand(HostName = virt2, SpmStatusVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:06:02,198-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler2) [2e8f57f9] Irs placed on server 'a0ccfd11-3402-4eba-a119-4987561b738e' failed. Proceed Failover 2017-10-01 16:06:02,255-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [2e8f57f9] START, ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 1700886e 2017-10-01 16:06:02,255-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [2e8f57f9] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:06:03,334-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [2e8f57f9] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:06:03,334-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [2e8f57f9] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:06:03,334-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [2e8f57f9] HostName = virt4 2017-10-01 16:06:03,334-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [2e8f57f9] Command 'ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:06:03,334-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [2e8f57f9] FINISH, ConnectStoragePoolVDSCommand, log id: 1700886e 2017-10-01 16:06:03,334-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler2) [2e8f57f9] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:06:04,453-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [2e8f57f9] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:06:04,453-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler2) [2e8f57f9] Command 'SpmStatusVDSCommand(HostName = virt4, SpmStatusVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:06:04,453-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler2) [2e8f57f9] Irs placed on server '5ae29ed2-9c0f-4754-8df3-b43d14726846' failed. Proceed Failover 2017-10-01 16:06:04,511-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [2e8f57f9] START, ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 4139ca26 2017-10-01 16:06:04,512-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [2e8f57f9] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:06:04,905-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [2e8f57f9] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:06:04,905-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [2e8f57f9] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:06:04,905-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [2e8f57f9] HostName = virt3 2017-10-01 16:06:04,905-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [2e8f57f9] Command 'ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:06:04,905-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [2e8f57f9] FINISH, ConnectStoragePoolVDSCommand, log id: 4139ca26 2017-10-01 16:06:04,905-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler2) [2e8f57f9] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:06:05,031-04 INFO [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-25) [4a3ca77c] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: c8653dba-63ec-4447-872c-0ffcc6af9c0f Type: Storage 2017-10-01 16:06:05,041-04 WARN [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-25) [4a3ca77c] skipping connect and refresh for all hosts, last master 'true', command status 'true' 2017-10-01 16:06:05,048-04 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-6-thread-29) [2e8f57f9] Finished reconstruct for pool 'aca61eac-ed8a-484b-bbee-3645148f6405'. Clearing event queue 2017-10-01 16:06:15,098-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [46f4ffda] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command HSMGetAllTasksStatusesVDS failed: Not SPM: () 2017-10-01 16:06:15,098-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler6) [46f4ffda] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand' return value ' 2017-10-01 16:06:15,098-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler6) [46f4ffda] HostName = virt3 2017-10-01 16:06:15,098-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler6) [46f4ffda] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = virt3, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736'})' execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () 2017-10-01 16:06:15,160-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [46f4ffda] START, ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 1a0f50ea 2017-10-01 16:06:15,160-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [46f4ffda] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:06:16,249-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [46f4ffda] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:06:16,249-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [46f4ffda] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:06:16,249-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [46f4ffda] HostName = virt2 2017-10-01 16:06:16,249-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [46f4ffda] Command 'ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:06:16,249-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [46f4ffda] FINISH, ConnectStoragePoolVDSCommand, log id: 1a0f50ea 2017-10-01 16:06:16,249-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler6) [46f4ffda] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:06:17,261-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [46f4ffda] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:06:17,261-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler6) [46f4ffda] Command 'SpmStatusVDSCommand(HostName = virt2, SpmStatusVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:06:17,261-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler6) [46f4ffda] Irs placed on server 'a0ccfd11-3402-4eba-a119-4987561b738e' failed. Proceed Failover 2017-10-01 16:06:17,318-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [46f4ffda] START, ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 5ffaad7a 2017-10-01 16:06:17,318-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [46f4ffda] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:06:18,396-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [46f4ffda] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:06:18,396-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [46f4ffda] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:06:18,396-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [46f4ffda] HostName = virt4 2017-10-01 16:06:18,396-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [46f4ffda] Command 'ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:06:18,396-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [46f4ffda] FINISH, ConnectStoragePoolVDSCommand, log id: 5ffaad7a 2017-10-01 16:06:18,396-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler6) [46f4ffda] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:06:19,515-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [46f4ffda] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:06:19,515-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler6) [46f4ffda] Command 'SpmStatusVDSCommand(HostName = virt4, SpmStatusVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:06:19,515-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler6) [46f4ffda] Irs placed on server '5ae29ed2-9c0f-4754-8df3-b43d14726846' failed. Proceed Failover 2017-10-01 16:06:19,572-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [46f4ffda] START, ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 2a154258 2017-10-01 16:06:19,573-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [46f4ffda] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:06:20,682-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [46f4ffda] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:06:20,682-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [46f4ffda] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:06:20,682-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [46f4ffda] HostName = virt3 2017-10-01 16:06:20,682-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [46f4ffda] Command 'ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:06:20,682-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [46f4ffda] FINISH, ConnectStoragePoolVDSCommand, log id: 2a154258 2017-10-01 16:06:20,682-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler6) [46f4ffda] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:06:20,997-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler6) [46f4ffda] Irs placed on server 'e1e71e9d-a0cd-47db-96c6-f724f6cea736' failed. Proceed Failover 2017-10-01 16:06:21,054-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [46f4ffda] START, ConnectStoragePoolVDSCommand(HostName = virt1, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vdsId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 4b236611 2017-10-01 16:06:21,054-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [46f4ffda] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:06:22,154-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [46f4ffda] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt1 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:06:22,154-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [46f4ffda] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:06:22,154-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [46f4ffda] HostName = virt1 2017-10-01 16:06:22,154-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [46f4ffda] Command 'ConnectStoragePoolVDSCommand(HostName = virt1, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vdsId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:06:22,154-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [46f4ffda] FINISH, ConnectStoragePoolVDSCommand, log id: 4b236611 2017-10-01 16:06:22,154-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler6) [46f4ffda] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:06:22,279-04 INFO [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-31) [21c3f543] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: c8653dba-63ec-4447-872c-0ffcc6af9c0f Type: Storage 2017-10-01 16:06:22,289-04 WARN [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-31) [21c3f543] skipping connect and refresh for all hosts, last master 'true', command status 'true' 2017-10-01 16:06:22,296-04 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-6-thread-16) [46f4ffda] Finished reconstruct for pool 'aca61eac-ed8a-484b-bbee-3645148f6405'. Clearing event queue 2017-10-01 16:06:32,326-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [59451f31] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt1 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:06:32,326-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler2) [59451f31] Command 'SpmStatusVDSCommand(HostName = virt1, SpmStatusVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:06:33,335-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [59451f31] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt1 command HSMGetAllTasksStatusesVDS failed: Not SPM 2017-10-01 16:06:33,335-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler2) [59451f31] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand' return value ' 2017-10-01 16:06:33,335-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler2) [59451f31] HostName = virt1 2017-10-01 16:06:33,335-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler2) [59451f31] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = virt1, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836'})' execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM 2017-10-01 16:06:33,396-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [59451f31] START, ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 7d2d3abb 2017-10-01 16:06:33,396-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [59451f31] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:06:34,503-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [59451f31] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:06:34,503-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [59451f31] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:06:34,503-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [59451f31] HostName = virt3 2017-10-01 16:06:34,503-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [59451f31] Command 'ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:06:34,503-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [59451f31] FINISH, ConnectStoragePoolVDSCommand, log id: 7d2d3abb 2017-10-01 16:06:34,503-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler2) [59451f31] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:06:35,616-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler2) [59451f31] Irs placed on server 'e1e71e9d-a0cd-47db-96c6-f724f6cea736' failed. Proceed Failover 2017-10-01 16:06:35,673-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [59451f31] START, ConnectStoragePoolVDSCommand(HostName = virt1, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vdsId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 69877ff1 2017-10-01 16:06:35,674-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [59451f31] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:06:36,150-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [59451f31] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt1 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:06:36,151-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [59451f31] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:06:36,151-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [59451f31] HostName = virt1 2017-10-01 16:06:36,151-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [59451f31] Command 'ConnectStoragePoolVDSCommand(HostName = virt1, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vdsId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:06:36,151-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [59451f31] FINISH, ConnectStoragePoolVDSCommand, log id: 69877ff1 2017-10-01 16:06:36,151-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler2) [59451f31] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:06:37,271-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [59451f31] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt1 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:06:37,271-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler2) [59451f31] Command 'SpmStatusVDSCommand(HostName = virt1, SpmStatusVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:06:37,271-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler2) [59451f31] Irs placed on server '45f8b331-842e-48e7-9df8-56adddb93836' failed. Proceed Failover 2017-10-01 16:06:37,328-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [59451f31] START, ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 6a59235b 2017-10-01 16:06:37,328-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [59451f31] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:06:38,419-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [59451f31] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:06:38,419-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [59451f31] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:06:38,419-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [59451f31] HostName = virt2 2017-10-01 16:06:38,419-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [59451f31] Command 'ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:06:38,419-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [59451f31] FINISH, ConnectStoragePoolVDSCommand, log id: 6a59235b 2017-10-01 16:06:38,419-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler2) [59451f31] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:06:39,526-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [59451f31] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:06:39,526-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler2) [59451f31] Command 'SpmStatusVDSCommand(HostName = virt2, SpmStatusVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:06:39,526-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler2) [59451f31] Irs placed on server 'a0ccfd11-3402-4eba-a119-4987561b738e' failed. Proceed Failover 2017-10-01 16:06:39,583-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [59451f31] START, ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 6e10f098 2017-10-01 16:06:39,583-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [59451f31] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:06:40,434-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [59451f31] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:06:40,434-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [59451f31] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:06:40,434-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [59451f31] HostName = virt4 2017-10-01 16:06:40,434-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [59451f31] Command 'ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:06:40,434-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [59451f31] FINISH, ConnectStoragePoolVDSCommand, log id: 6e10f098 2017-10-01 16:06:40,434-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler2) [59451f31] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:06:40,560-04 INFO [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-4) [5e2ac6c2] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: c8653dba-63ec-4447-872c-0ffcc6af9c0f Type: Storage 2017-10-01 16:06:40,574-04 WARN [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-4) [5e2ac6c2] skipping connect and refresh for all hosts, last master 'true', command status 'true' 2017-10-01 16:06:40,591-04 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-6-thread-21) [59451f31] Finished reconstruct for pool 'aca61eac-ed8a-484b-bbee-3645148f6405'. Clearing event queue 2017-10-01 16:06:46,867-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-50) [] starting processDomainRecovery for domain 'f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2:export'. 2017-10-01 16:06:46,921-04 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-50) [] Storage domain 'f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2:export' is not visible to one or more hosts. Since the domain's type is 'ImportExport', hosts status will not be changed to non-operational 2017-10-01 16:06:46,922-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-50) [] Removing vds '[]' from the domain in maintenance cache 2017-10-01 16:06:46,922-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-50) [] Removing host(s) '[]' from hosts unseen domain report cache 2017-10-01 16:06:51,351-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [11c50e6e] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:06:51,351-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler8) [11c50e6e] Command 'SpmStatusVDSCommand(HostName = virt4, SpmStatusVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:06:51,374-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [11c50e6e] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command HSMGetAllTasksStatusesVDS failed: Not SPM 2017-10-01 16:06:51,374-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler8) [11c50e6e] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand' return value ' 2017-10-01 16:06:51,374-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler8) [11c50e6e] HostName = virt4 2017-10-01 16:06:51,374-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler8) [11c50e6e] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = virt4, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846'})' execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM 2017-10-01 16:06:51,435-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [11c50e6e] START, ConnectStoragePoolVDSCommand(HostName = virt1, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vdsId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 7be3e711 2017-10-01 16:06:51,435-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [11c50e6e] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:06:52,534-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [11c50e6e] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt1 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:06:52,534-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [11c50e6e] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:06:52,534-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [11c50e6e] HostName = virt1 2017-10-01 16:06:52,534-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [11c50e6e] Command 'ConnectStoragePoolVDSCommand(HostName = virt1, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vdsId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:06:52,534-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [11c50e6e] FINISH, ConnectStoragePoolVDSCommand, log id: 7be3e711 2017-10-01 16:06:52,534-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler8) [11c50e6e] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:06:53,654-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [11c50e6e] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt1 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:06:53,654-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler8) [11c50e6e] Command 'SpmStatusVDSCommand(HostName = virt1, SpmStatusVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:06:53,654-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler8) [11c50e6e] Irs placed on server '45f8b331-842e-48e7-9df8-56adddb93836' failed. Proceed Failover 2017-10-01 16:06:53,712-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [11c50e6e] START, ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 5866d71c 2017-10-01 16:06:53,712-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [11c50e6e] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:06:54,675-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [11c50e6e] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:06:54,675-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [11c50e6e] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:06:54,675-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [11c50e6e] HostName = virt2 2017-10-01 16:06:54,675-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [11c50e6e] Command 'ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:06:54,675-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [11c50e6e] FINISH, ConnectStoragePoolVDSCommand, log id: 5866d71c 2017-10-01 16:06:54,675-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler8) [11c50e6e] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:06:55,795-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [11c50e6e] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:06:55,795-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler8) [11c50e6e] Command 'SpmStatusVDSCommand(HostName = virt2, SpmStatusVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:06:55,795-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler8) [11c50e6e] Irs placed on server 'a0ccfd11-3402-4eba-a119-4987561b738e' failed. Proceed Failover 2017-10-01 16:06:55,853-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [11c50e6e] START, ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 6985fb70 2017-10-01 16:06:55,853-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [11c50e6e] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:06:56,333-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [617e13be] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 16:06:56,348-04 INFO [org.ovirt.engine.core.sso.servlets.OAuthRevokeServlet] (default task-4) [] User admin@internal successfully logged out 2017-10-01 16:06:56,363-04 INFO [org.ovirt.engine.core.bll.aaa.TerminateSessionsForTokenCommand] (default task-10) [225b5cb8] Running command: TerminateSessionsForTokenCommand internal: true. 2017-10-01 16:06:56,438-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [11c50e6e] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:06:56,438-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [11c50e6e] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:06:56,438-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [11c50e6e] HostName = virt3 2017-10-01 16:06:56,438-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [11c50e6e] Command 'ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:06:56,438-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [11c50e6e] FINISH, ConnectStoragePoolVDSCommand, log id: 6985fb70 2017-10-01 16:06:56,438-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler8) [11c50e6e] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:06:57,551-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler8) [11c50e6e] Irs placed on server 'e1e71e9d-a0cd-47db-96c6-f724f6cea736' failed. Proceed Failover 2017-10-01 16:06:57,615-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [11c50e6e] START, ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 589121e3 2017-10-01 16:06:57,615-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [11c50e6e] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:06:58,197-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [11c50e6e] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:06:58,197-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [11c50e6e] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:06:58,197-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [11c50e6e] HostName = virt4 2017-10-01 16:06:58,197-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [11c50e6e] Command 'ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:06:58,197-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [11c50e6e] FINISH, ConnectStoragePoolVDSCommand, log id: 589121e3 2017-10-01 16:06:58,197-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler8) [11c50e6e] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:06:58,337-04 INFO [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-47) [985bd3] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: c8653dba-63ec-4447-872c-0ffcc6af9c0f Type: Storage 2017-10-01 16:06:58,349-04 WARN [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-47) [985bd3] skipping connect and refresh for all hosts, last master 'true', command status 'true' 2017-10-01 16:06:58,364-04 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-6-thread-14) [11c50e6e] Finished reconstruct for pool 'aca61eac-ed8a-484b-bbee-3645148f6405'. Clearing event queue 2017-10-01 16:06:59,172-04 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-2) [] domain 'f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2:export' in problem 'NOT_REPORTED'. vds: 'virt3' 2017-10-01 16:07:01,930-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-49) [] starting processDomainRecovery for domain '63d680c7-4c94-4b29-a6cb-f288a128b9bc:iso'. 2017-10-01 16:07:01,986-04 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-49) [] Storage domain '63d680c7-4c94-4b29-a6cb-f288a128b9bc:iso' is not visible to one or more hosts. Since the domain's type is 'ISO', hosts status will not be changed to non-operational 2017-10-01 16:07:01,986-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-49) [] Removing vds '[]' from the domain in maintenance cache 2017-10-01 16:07:01,986-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-49) [] Removing host(s) '[]' from hosts unseen domain report cache 2017-10-01 16:07:09,325-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [16c0bb48] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:07:09,325-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler8) [16c0bb48] Command 'SpmStatusVDSCommand(HostName = virt4, SpmStatusVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:07:09,894-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [16c0bb48] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command HSMGetAllTasksStatusesVDS failed: Not SPM 2017-10-01 16:07:09,894-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler8) [16c0bb48] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand' return value ' 2017-10-01 16:07:09,894-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler8) [16c0bb48] HostName = virt4 2017-10-01 16:07:09,894-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler8) [16c0bb48] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = virt4, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846'})' execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM 2017-10-01 16:07:09,956-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [16c0bb48] START, ConnectStoragePoolVDSCommand(HostName = virt1, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vdsId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 7448da60 2017-10-01 16:07:09,957-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [16c0bb48] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:07:09,968-04 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-9) [] domain '63d680c7-4c94-4b29-a6cb-f288a128b9bc:iso' in problem 'PROBLEMATIC'. vds: 'virt1' 2017-10-01 16:07:11,054-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [16c0bb48] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt1 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:07:11,054-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [16c0bb48] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:07:11,054-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [16c0bb48] HostName = virt1 2017-10-01 16:07:11,054-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [16c0bb48] Command 'ConnectStoragePoolVDSCommand(HostName = virt1, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vdsId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:07:11,054-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [16c0bb48] FINISH, ConnectStoragePoolVDSCommand, log id: 7448da60 2017-10-01 16:07:11,054-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler8) [16c0bb48] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:07:11,556-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [16c0bb48] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt1 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:07:11,556-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler8) [16c0bb48] Command 'SpmStatusVDSCommand(HostName = virt1, SpmStatusVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:07:11,556-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler8) [16c0bb48] Irs placed on server '45f8b331-842e-48e7-9df8-56adddb93836' failed. Proceed Failover 2017-10-01 16:07:11,613-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [16c0bb48] START, ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 6f089a44 2017-10-01 16:07:11,613-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [16c0bb48] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:07:12,705-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [16c0bb48] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:07:12,705-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [16c0bb48] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:07:12,705-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [16c0bb48] HostName = virt2 2017-10-01 16:07:12,705-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [16c0bb48] Command 'ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:07:12,705-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [16c0bb48] FINISH, ConnectStoragePoolVDSCommand, log id: 6f089a44 2017-10-01 16:07:12,705-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler8) [16c0bb48] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:07:13,825-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [16c0bb48] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:07:13,825-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler8) [16c0bb48] Command 'SpmStatusVDSCommand(HostName = virt2, SpmStatusVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:07:13,825-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler8) [16c0bb48] Irs placed on server 'a0ccfd11-3402-4eba-a119-4987561b738e' failed. Proceed Failover 2017-10-01 16:07:13,889-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [16c0bb48] START, ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 51bd194e 2017-10-01 16:07:13,889-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [16c0bb48] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:07:14,254-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [16c0bb48] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:07:14,254-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [16c0bb48] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:07:14,254-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [16c0bb48] HostName = virt4 2017-10-01 16:07:14,254-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [16c0bb48] Command 'ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:07:14,254-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [16c0bb48] FINISH, ConnectStoragePoolVDSCommand, log id: 51bd194e 2017-10-01 16:07:14,254-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler8) [16c0bb48] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:07:15,169-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [16c0bb48] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:07:15,169-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler8) [16c0bb48] Command 'SpmStatusVDSCommand(HostName = virt4, SpmStatusVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:07:15,169-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler8) [16c0bb48] Irs placed on server '5ae29ed2-9c0f-4754-8df3-b43d14726846' failed. Proceed Failover 2017-10-01 16:07:15,226-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [16c0bb48] START, ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 838222c 2017-10-01 16:07:15,227-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [16c0bb48] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:07:22,192-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [16c0bb48] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:07:22,192-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [16c0bb48] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:07:22,192-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [16c0bb48] HostName = virt3 2017-10-01 16:07:22,192-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [16c0bb48] Command 'ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:07:22,192-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [16c0bb48] FINISH, ConnectStoragePoolVDSCommand, log id: 838222c 2017-10-01 16:07:22,192-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler8) [16c0bb48] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:07:22,324-04 INFO [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-13) [7bd370ba] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: c8653dba-63ec-4447-872c-0ffcc6af9c0f Type: Storage 2017-10-01 16:07:22,335-04 WARN [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-13) [7bd370ba] skipping connect and refresh for all hosts, last master 'true', command status 'true' 2017-10-01 16:07:22,348-04 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-6-thread-19) [16c0bb48] Finished reconstruct for pool 'aca61eac-ed8a-484b-bbee-3645148f6405'. Clearing event queue 2017-10-01 16:07:34,337-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [7748171b] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command HSMGetAllTasksStatusesVDS failed: Not SPM: () 2017-10-01 16:07:34,337-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler1) [7748171b] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand' return value ' 2017-10-01 16:07:34,337-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler1) [7748171b] HostName = virt3 2017-10-01 16:07:34,337-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler1) [7748171b] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = virt3, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736'})' execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () 2017-10-01 16:07:34,472-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [7748171b] START, ConnectStoragePoolVDSCommand(HostName = virt1, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vdsId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 67c199df 2017-10-01 16:07:34,472-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [7748171b] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:07:35,351-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [7748171b] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt1 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:07:35,351-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [7748171b] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:07:35,351-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [7748171b] HostName = virt1 2017-10-01 16:07:35,351-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [7748171b] Command 'ConnectStoragePoolVDSCommand(HostName = virt1, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vdsId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:07:35,351-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [7748171b] FINISH, ConnectStoragePoolVDSCommand, log id: 67c199df 2017-10-01 16:07:35,351-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler1) [7748171b] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:07:36,483-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [7748171b] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt1 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:07:36,483-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler1) [7748171b] Command 'SpmStatusVDSCommand(HostName = virt1, SpmStatusVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:07:36,483-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler1) [7748171b] Irs placed on server '45f8b331-842e-48e7-9df8-56adddb93836' failed. Proceed Failover 2017-10-01 16:07:36,540-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [7748171b] START, ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 16591c22 2017-10-01 16:07:36,541-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [7748171b] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:07:37,618-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [7748171b] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:07:37,618-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [7748171b] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:07:37,618-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [7748171b] HostName = virt4 2017-10-01 16:07:37,618-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [7748171b] Command 'ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:07:37,618-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [7748171b] FINISH, ConnectStoragePoolVDSCommand, log id: 16591c22 2017-10-01 16:07:37,618-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler1) [7748171b] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:07:38,738-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [7748171b] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:07:38,739-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler1) [7748171b] Command 'SpmStatusVDSCommand(HostName = virt4, SpmStatusVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:07:38,739-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler1) [7748171b] Irs placed on server '5ae29ed2-9c0f-4754-8df3-b43d14726846' failed. Proceed Failover 2017-10-01 16:07:38,809-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [7748171b] START, ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 21634d73 2017-10-01 16:07:38,809-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [7748171b] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:07:39,898-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [7748171b] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:07:39,898-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [7748171b] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:07:39,898-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [7748171b] HostName = virt2 2017-10-01 16:07:39,898-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [7748171b] Command 'ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:07:39,898-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [7748171b] FINISH, ConnectStoragePoolVDSCommand, log id: 21634d73 2017-10-01 16:07:39,898-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler1) [7748171b] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:07:41,018-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [7748171b] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:07:41,018-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler1) [7748171b] Command 'SpmStatusVDSCommand(HostName = virt2, SpmStatusVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:07:41,018-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler1) [7748171b] Irs placed on server 'a0ccfd11-3402-4eba-a119-4987561b738e' failed. Proceed Failover 2017-10-01 16:07:41,076-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [7748171b] START, ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 2aa866e6 2017-10-01 16:07:41,076-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [7748171b] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:07:42,185-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [7748171b] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:07:42,185-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [7748171b] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:07:42,185-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [7748171b] HostName = virt3 2017-10-01 16:07:42,185-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [7748171b] Command 'ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:07:42,185-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [7748171b] FINISH, ConnectStoragePoolVDSCommand, log id: 2aa866e6 2017-10-01 16:07:42,185-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler1) [7748171b] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:07:42,318-04 INFO [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-42) [6712a30a] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: c8653dba-63ec-4447-872c-0ffcc6af9c0f Type: Storage 2017-10-01 16:07:42,387-04 WARN [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-42) [6712a30a] skipping connect and refresh for all hosts, last master 'true', command status 'true' 2017-10-01 16:07:42,394-04 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-6-thread-12) [7748171b] Finished reconstruct for pool 'aca61eac-ed8a-484b-bbee-3645148f6405'. Clearing event queue 2017-10-01 16:07:53,902-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [4647245c] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command HSMGetAllTasksStatusesVDS failed: Not SPM: () 2017-10-01 16:07:53,902-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler4) [4647245c] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand' return value ' 2017-10-01 16:07:53,903-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler4) [4647245c] HostName = virt3 2017-10-01 16:07:53,903-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler4) [4647245c] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = virt3, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736'})' execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () 2017-10-01 16:07:53,966-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [4647245c] START, ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 64a40b79 2017-10-01 16:07:53,966-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [4647245c] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:07:55,055-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [4647245c] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:07:55,055-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [4647245c] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:07:55,055-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [4647245c] HostName = virt2 2017-10-01 16:07:55,055-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [4647245c] Command 'ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:07:55,055-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [4647245c] FINISH, ConnectStoragePoolVDSCommand, log id: 64a40b79 2017-10-01 16:07:55,055-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler4) [4647245c] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:07:56,175-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [4647245c] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:07:56,175-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler4) [4647245c] Command 'SpmStatusVDSCommand(HostName = virt2, SpmStatusVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:07:56,175-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler4) [4647245c] Irs placed on server 'a0ccfd11-3402-4eba-a119-4987561b738e' failed. Proceed Failover 2017-10-01 16:07:56,232-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [4647245c] START, ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 6c153651 2017-10-01 16:07:56,232-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [4647245c] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:07:56,394-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [6001fc60] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 16:07:57,340-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [4647245c] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:07:57,340-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [4647245c] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:07:57,340-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [4647245c] HostName = virt3 2017-10-01 16:07:57,340-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [4647245c] Command 'ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:07:57,340-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [4647245c] FINISH, ConnectStoragePoolVDSCommand, log id: 6c153651 2017-10-01 16:07:57,340-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler4) [4647245c] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:07:58,453-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler4) [4647245c] Irs placed on server 'e1e71e9d-a0cd-47db-96c6-f724f6cea736' failed. Proceed Failover 2017-10-01 16:07:58,511-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [4647245c] START, ConnectStoragePoolVDSCommand(HostName = virt1, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vdsId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 6a14b8c2 2017-10-01 16:07:58,511-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [4647245c] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:07:58,629-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [4647245c] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt1 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:07:58,629-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [4647245c] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:07:58,629-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [4647245c] HostName = virt1 2017-10-01 16:07:58,629-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [4647245c] Command 'ConnectStoragePoolVDSCommand(HostName = virt1, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vdsId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:07:58,629-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [4647245c] FINISH, ConnectStoragePoolVDSCommand, log id: 6a14b8c2 2017-10-01 16:07:58,629-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler4) [4647245c] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:07:59,749-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [4647245c] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt1 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:07:59,749-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler4) [4647245c] Command 'SpmStatusVDSCommand(HostName = virt1, SpmStatusVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:07:59,750-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler4) [4647245c] Irs placed on server '45f8b331-842e-48e7-9df8-56adddb93836' failed. Proceed Failover 2017-10-01 16:07:59,807-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [4647245c] START, ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 537ffa48 2017-10-01 16:07:59,807-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [4647245c] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:08:00,884-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [4647245c] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:08:00,884-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [4647245c] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:08:00,884-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [4647245c] HostName = virt4 2017-10-01 16:08:00,884-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [4647245c] Command 'ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:08:00,884-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [4647245c] FINISH, ConnectStoragePoolVDSCommand, log id: 537ffa48 2017-10-01 16:08:00,884-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler4) [4647245c] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:08:01,010-04 INFO [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-6) [e6ec13d] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: c8653dba-63ec-4447-872c-0ffcc6af9c0f Type: Storage 2017-10-01 16:08:01,022-04 WARN [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-6) [e6ec13d] skipping connect and refresh for all hosts, last master 'true', command status 'true' 2017-10-01 16:08:01,029-04 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-6-thread-35) [4647245c] Finished reconstruct for pool 'aca61eac-ed8a-484b-bbee-3645148f6405'. Clearing event queue 2017-10-01 16:08:11,426-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [f234495] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:08:11,426-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler3) [f234495] Command 'SpmStatusVDSCommand(HostName = virt4, SpmStatusVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:08:11,438-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [f234495] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command HSMGetAllTasksStatusesVDS failed: Not SPM 2017-10-01 16:08:11,438-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler3) [f234495] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand' return value ' 2017-10-01 16:08:11,438-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler3) [f234495] HostName = virt4 2017-10-01 16:08:11,438-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler3) [f234495] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = virt4, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846'})' execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM 2017-10-01 16:08:11,501-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [f234495] START, ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 40642569 2017-10-01 16:08:11,501-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [f234495] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:08:12,577-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [f234495] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:08:12,577-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [f234495] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:08:12,577-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [f234495] HostName = virt4 2017-10-01 16:08:12,577-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [f234495] Command 'ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:08:12,577-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [f234495] FINISH, ConnectStoragePoolVDSCommand, log id: 40642569 2017-10-01 16:08:12,577-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler3) [f234495] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:08:13,695-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [f234495] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:08:13,695-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler3) [f234495] Command 'SpmStatusVDSCommand(HostName = virt4, SpmStatusVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:08:13,695-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler3) [f234495] Irs placed on server '5ae29ed2-9c0f-4754-8df3-b43d14726846' failed. Proceed Failover 2017-10-01 16:08:13,753-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [f234495] START, ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 5ad9e7e6 2017-10-01 16:08:13,753-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [f234495] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:08:14,929-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [f234495] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:08:14,929-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [f234495] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:08:14,929-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [f234495] HostName = virt2 2017-10-01 16:08:14,929-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [f234495] Command 'ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:08:14,929-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [f234495] FINISH, ConnectStoragePoolVDSCommand, log id: 5ad9e7e6 2017-10-01 16:08:14,929-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler3) [f234495] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:08:16,049-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [f234495] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:08:16,049-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler3) [f234495] Command 'SpmStatusVDSCommand(HostName = virt2, SpmStatusVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:08:16,049-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler3) [f234495] Irs placed on server 'a0ccfd11-3402-4eba-a119-4987561b738e' failed. Proceed Failover 2017-10-01 16:08:16,106-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [f234495] START, ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 470572fb 2017-10-01 16:08:16,107-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [f234495] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:08:17,214-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [f234495] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:08:17,214-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [f234495] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:08:17,214-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [f234495] HostName = virt3 2017-10-01 16:08:17,214-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [f234495] Command 'ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:08:17,214-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [f234495] FINISH, ConnectStoragePoolVDSCommand, log id: 470572fb 2017-10-01 16:08:17,214-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler3) [f234495] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:08:17,345-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler3) [f234495] Irs placed on server 'e1e71e9d-a0cd-47db-96c6-f724f6cea736' failed. Proceed Failover 2017-10-01 16:08:17,403-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [f234495] START, ConnectStoragePoolVDSCommand(HostName = virt1, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vdsId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 19cf8824 2017-10-01 16:08:17,403-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [f234495] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:08:18,246-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [f234495] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt1 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:08:18,246-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [f234495] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:08:18,246-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [f234495] HostName = virt1 2017-10-01 16:08:18,246-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [f234495] Command 'ConnectStoragePoolVDSCommand(HostName = virt1, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vdsId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:08:18,246-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [f234495] FINISH, ConnectStoragePoolVDSCommand, log id: 19cf8824 2017-10-01 16:08:18,246-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler3) [f234495] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:08:18,373-04 INFO [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-48) [4931d8fc] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: c8653dba-63ec-4447-872c-0ffcc6af9c0f Type: Storage 2017-10-01 16:08:18,387-04 WARN [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-48) [4931d8fc] skipping connect and refresh for all hosts, last master 'true', command status 'true' 2017-10-01 16:08:18,396-04 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-6-thread-2) [f234495] Finished reconstruct for pool 'aca61eac-ed8a-484b-bbee-3645148f6405'. Clearing event queue 2017-10-01 16:08:28,759-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [599b4232] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt1 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:08:28,759-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler5) [599b4232] Command 'SpmStatusVDSCommand(HostName = virt1, SpmStatusVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:08:29,769-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [599b4232] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt1 command HSMGetAllTasksStatusesVDS failed: Not SPM 2017-10-01 16:08:29,769-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler5) [599b4232] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand' return value ' 2017-10-01 16:08:29,769-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler5) [599b4232] HostName = virt1 2017-10-01 16:08:29,769-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler5) [599b4232] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = virt1, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836'})' execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM 2017-10-01 16:08:29,831-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [599b4232] START, ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 21b4623b 2017-10-01 16:08:29,831-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [599b4232] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:08:30,908-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [599b4232] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:08:30,908-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [599b4232] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:08:30,908-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [599b4232] HostName = virt4 2017-10-01 16:08:30,908-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [599b4232] Command 'ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:08:30,908-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [599b4232] FINISH, ConnectStoragePoolVDSCommand, log id: 21b4623b 2017-10-01 16:08:30,908-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler5) [599b4232] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:08:32,028-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [599b4232] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:08:32,028-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler5) [599b4232] Command 'SpmStatusVDSCommand(HostName = virt4, SpmStatusVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:08:32,028-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler5) [599b4232] Irs placed on server '5ae29ed2-9c0f-4754-8df3-b43d14726846' failed. Proceed Failover 2017-10-01 16:08:32,086-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [599b4232] START, ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 377037cc 2017-10-01 16:08:32,086-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [599b4232] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:08:32,497-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [599b4232] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:08:32,497-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [599b4232] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:08:32,497-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [599b4232] HostName = virt2 2017-10-01 16:08:32,497-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [599b4232] Command 'ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:08:32,497-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [599b4232] FINISH, ConnectStoragePoolVDSCommand, log id: 377037cc 2017-10-01 16:08:32,497-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler5) [599b4232] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:08:33,619-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [599b4232] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:08:33,619-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler5) [599b4232] Command 'SpmStatusVDSCommand(HostName = virt2, SpmStatusVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:08:33,619-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler5) [599b4232] Irs placed on server 'a0ccfd11-3402-4eba-a119-4987561b738e' failed. Proceed Failover 2017-10-01 16:08:33,677-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [599b4232] START, ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 55661dd4 2017-10-01 16:08:33,677-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [599b4232] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:08:34,249-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [599b4232] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:08:34,249-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [599b4232] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:08:34,249-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [599b4232] HostName = virt3 2017-10-01 16:08:34,249-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [599b4232] Command 'ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:08:34,249-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [599b4232] FINISH, ConnectStoragePoolVDSCommand, log id: 55661dd4 2017-10-01 16:08:34,249-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler5) [599b4232] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:08:35,329-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler5) [599b4232] Irs placed on server 'e1e71e9d-a0cd-47db-96c6-f724f6cea736' failed. Proceed Failover 2017-10-01 16:08:35,386-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [599b4232] START, ConnectStoragePoolVDSCommand(HostName = virt1, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vdsId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 2b42689d 2017-10-01 16:08:35,386-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [599b4232] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:08:36,483-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [599b4232] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt1 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:08:36,483-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [599b4232] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:08:36,483-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [599b4232] HostName = virt1 2017-10-01 16:08:36,483-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [599b4232] Command 'ConnectStoragePoolVDSCommand(HostName = virt1, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vdsId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:08:36,483-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [599b4232] FINISH, ConnectStoragePoolVDSCommand, log id: 2b42689d 2017-10-01 16:08:36,483-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler5) [599b4232] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:08:36,626-04 INFO [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-26) [2bb058f7] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: c8653dba-63ec-4447-872c-0ffcc6af9c0f Type: Storage 2017-10-01 16:08:36,639-04 WARN [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-26) [2bb058f7] skipping connect and refresh for all hosts, last master 'true', command status 'true' 2017-10-01 16:08:36,647-04 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-6-thread-46) [599b4232] Finished reconstruct for pool 'aca61eac-ed8a-484b-bbee-3645148f6405'. Clearing event queue 2017-10-01 16:08:47,608-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [32e6d7ad] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt1 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:08:47,609-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler9) [32e6d7ad] Command 'SpmStatusVDSCommand(HostName = virt1, SpmStatusVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:08:47,709-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [32e6d7ad] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt1 command HSMGetAllTasksStatusesVDS failed: Not SPM 2017-10-01 16:08:47,709-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler9) [32e6d7ad] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand' return value ' 2017-10-01 16:08:47,709-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler9) [32e6d7ad] HostName = virt1 2017-10-01 16:08:47,709-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler9) [32e6d7ad] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = virt1, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836'})' execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM 2017-10-01 16:08:47,771-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [32e6d7ad] START, ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 67505ef 2017-10-01 16:08:47,771-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [32e6d7ad] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:08:48,848-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [32e6d7ad] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:08:48,848-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [32e6d7ad] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:08:48,848-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [32e6d7ad] HostName = virt4 2017-10-01 16:08:48,848-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [32e6d7ad] Command 'ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:08:48,848-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [32e6d7ad] FINISH, ConnectStoragePoolVDSCommand, log id: 67505ef 2017-10-01 16:08:48,848-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler9) [32e6d7ad] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:08:49,364-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [32e6d7ad] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:08:49,364-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler9) [32e6d7ad] Command 'SpmStatusVDSCommand(HostName = virt4, SpmStatusVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:08:49,364-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler9) [32e6d7ad] Irs placed on server '5ae29ed2-9c0f-4754-8df3-b43d14726846' failed. Proceed Failover 2017-10-01 16:08:49,422-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [32e6d7ad] START, ConnectStoragePoolVDSCommand(HostName = virt1, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vdsId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 37b83e30 2017-10-01 16:08:49,422-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [32e6d7ad] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:08:50,395-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [32e6d7ad] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt1 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:08:50,395-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [32e6d7ad] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:08:50,395-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [32e6d7ad] HostName = virt1 2017-10-01 16:08:50,396-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [32e6d7ad] Command 'ConnectStoragePoolVDSCommand(HostName = virt1, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vdsId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:08:50,396-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [32e6d7ad] FINISH, ConnectStoragePoolVDSCommand, log id: 37b83e30 2017-10-01 16:08:50,396-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler9) [32e6d7ad] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:08:51,516-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [32e6d7ad] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt1 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:08:51,516-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler9) [32e6d7ad] Command 'SpmStatusVDSCommand(HostName = virt1, SpmStatusVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:08:51,516-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler9) [32e6d7ad] Irs placed on server '45f8b331-842e-48e7-9df8-56adddb93836' failed. Proceed Failover 2017-10-01 16:08:51,573-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [32e6d7ad] START, ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 4c3b252e 2017-10-01 16:08:51,573-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [32e6d7ad] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:08:52,662-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [32e6d7ad] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:08:52,663-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [32e6d7ad] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:08:52,663-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [32e6d7ad] HostName = virt2 2017-10-01 16:08:52,663-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [32e6d7ad] Command 'ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:08:52,663-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [32e6d7ad] FINISH, ConnectStoragePoolVDSCommand, log id: 4c3b252e 2017-10-01 16:08:52,663-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler9) [32e6d7ad] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:08:53,583-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [32e6d7ad] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:08:53,583-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler9) [32e6d7ad] Command 'SpmStatusVDSCommand(HostName = virt2, SpmStatusVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:08:53,583-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler9) [32e6d7ad] Irs placed on server 'a0ccfd11-3402-4eba-a119-4987561b738e' failed. Proceed Failover 2017-10-01 16:08:53,640-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [32e6d7ad] START, ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 50669367 2017-10-01 16:08:53,640-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [32e6d7ad] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:08:56,414-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [59a87ccd] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 16:08:57,348-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [32e6d7ad] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:08:57,348-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [32e6d7ad] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:08:57,348-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [32e6d7ad] HostName = virt3 2017-10-01 16:08:57,348-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [32e6d7ad] Command 'ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:08:57,348-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [32e6d7ad] FINISH, ConnectStoragePoolVDSCommand, log id: 50669367 2017-10-01 16:08:57,348-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler9) [32e6d7ad] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:08:57,477-04 INFO [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-31) [6c03736b] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: c8653dba-63ec-4447-872c-0ffcc6af9c0f Type: Storage 2017-10-01 16:08:57,490-04 WARN [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-31) [6c03736b] skipping connect and refresh for all hosts, last master 'true', command status 'true' 2017-10-01 16:08:57,506-04 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-6-thread-16) [32e6d7ad] Finished reconstruct for pool 'aca61eac-ed8a-484b-bbee-3645148f6405'. Clearing event queue 2017-10-01 16:09:01,096-04 WARN [org.ovirt.engine.core.bll.storage.pool.ForceSelectSPMCommand] (default task-54) [fce4cf29-15c4-47ad-ab0b-8724afa2d639] Validation of action 'ForceSelectSPM' failed for user admin@internal. Reasons: VAR__ACTION__FORCE_SELECT,VAR__TYPE__SPM,$VdsName virt1,ACTION_TYPE_FAILED_IMAGE_REPOSITORY_NOT_FOUND 2017-10-01 16:09:08,742-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [64d2e2f3] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command HSMGetAllTasksStatusesVDS failed: Not SPM: () 2017-10-01 16:09:08,742-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler1) [64d2e2f3] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand' return value ' 2017-10-01 16:09:08,742-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler1) [64d2e2f3] HostName = virt3 2017-10-01 16:09:08,742-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler1) [64d2e2f3] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = virt3, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736'})' execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () 2017-10-01 16:09:08,815-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [64d2e2f3] START, ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 2a8397dd 2017-10-01 16:09:08,815-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [64d2e2f3] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:09:09,893-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [64d2e2f3] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:09:09,893-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [64d2e2f3] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:09:09,893-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [64d2e2f3] HostName = virt4 2017-10-01 16:09:09,893-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [64d2e2f3] Command 'ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:09:09,893-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [64d2e2f3] FINISH, ConnectStoragePoolVDSCommand, log id: 2a8397dd 2017-10-01 16:09:09,893-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler1) [64d2e2f3] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:09:10,502-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [64d2e2f3] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:09:10,502-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler1) [64d2e2f3] Command 'SpmStatusVDSCommand(HostName = virt4, SpmStatusVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:09:10,502-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler1) [64d2e2f3] Irs placed on server '5ae29ed2-9c0f-4754-8df3-b43d14726846' failed. Proceed Failover 2017-10-01 16:09:10,560-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [64d2e2f3] START, ConnectStoragePoolVDSCommand(HostName = virt1, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vdsId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 9ad561a 2017-10-01 16:09:10,560-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [64d2e2f3] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:09:11,659-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [64d2e2f3] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt1 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:09:11,659-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [64d2e2f3] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:09:11,659-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [64d2e2f3] HostName = virt1 2017-10-01 16:09:11,659-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [64d2e2f3] Command 'ConnectStoragePoolVDSCommand(HostName = virt1, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vdsId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:09:11,659-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [64d2e2f3] FINISH, ConnectStoragePoolVDSCommand, log id: 9ad561a 2017-10-01 16:09:11,659-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler1) [64d2e2f3] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:09:12,446-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [64d2e2f3] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt1 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:09:12,446-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler1) [64d2e2f3] Command 'SpmStatusVDSCommand(HostName = virt1, SpmStatusVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:09:12,446-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler1) [64d2e2f3] Irs placed on server '45f8b331-842e-48e7-9df8-56adddb93836' failed. Proceed Failover 2017-10-01 16:09:12,503-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [64d2e2f3] START, ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 21b5ce86 2017-10-01 16:09:12,503-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [64d2e2f3] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:09:13,597-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [64d2e2f3] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:09:13,597-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [64d2e2f3] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:09:13,597-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [64d2e2f3] HostName = virt2 2017-10-01 16:09:13,597-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [64d2e2f3] Command 'ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:09:13,597-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [64d2e2f3] FINISH, ConnectStoragePoolVDSCommand, log id: 21b5ce86 2017-10-01 16:09:13,597-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler1) [64d2e2f3] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:09:14,719-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [64d2e2f3] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:09:14,719-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler1) [64d2e2f3] Command 'SpmStatusVDSCommand(HostName = virt2, SpmStatusVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:09:14,719-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler1) [64d2e2f3] Irs placed on server 'a0ccfd11-3402-4eba-a119-4987561b738e' failed. Proceed Failover 2017-10-01 16:09:14,776-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [64d2e2f3] START, ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 67c0bd13 2017-10-01 16:09:14,777-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [64d2e2f3] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:09:15,779-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [64d2e2f3] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:09:15,780-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [64d2e2f3] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:09:15,780-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [64d2e2f3] HostName = virt3 2017-10-01 16:09:15,780-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [64d2e2f3] Command 'ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:09:15,780-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [64d2e2f3] FINISH, ConnectStoragePoolVDSCommand, log id: 67c0bd13 2017-10-01 16:09:15,780-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler1) [64d2e2f3] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:09:15,906-04 INFO [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-4) [7bcd7262] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: c8653dba-63ec-4447-872c-0ffcc6af9c0f Type: Storage 2017-10-01 16:09:15,914-04 WARN [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-4) [7bcd7262] skipping connect and refresh for all hosts, last master 'true', command status 'true' 2017-10-01 16:09:15,923-04 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-6-thread-21) [64d2e2f3] Finished reconstruct for pool 'aca61eac-ed8a-484b-bbee-3645148f6405'. Clearing event queue 2017-10-01 16:09:27,606-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [62b74618] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command HSMGetAllTasksStatusesVDS failed: Not SPM: () 2017-10-01 16:09:27,606-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler8) [62b74618] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand' return value ' 2017-10-01 16:09:27,606-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler8) [62b74618] HostName = virt3 2017-10-01 16:09:27,606-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler8) [62b74618] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = virt3, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736'})' execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () 2017-10-01 16:09:27,739-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [62b74618] START, ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 3d6b3b52 2017-10-01 16:09:27,739-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [62b74618] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:09:31,614-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [62b74618] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:09:31,614-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [62b74618] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:09:31,614-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [62b74618] HostName = virt2 2017-10-01 16:09:31,614-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [62b74618] Command 'ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:09:31,614-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [62b74618] FINISH, ConnectStoragePoolVDSCommand, log id: 3d6b3b52 2017-10-01 16:09:31,614-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler8) [62b74618] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:09:31,782-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [62b74618] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:09:31,782-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler8) [62b74618] Command 'SpmStatusVDSCommand(HostName = virt2, SpmStatusVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:09:31,782-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler8) [62b74618] Irs placed on server 'a0ccfd11-3402-4eba-a119-4987561b738e' failed. Proceed Failover 2017-10-01 16:09:31,839-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [62b74618] START, ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 3b5bd5f7 2017-10-01 16:09:31,839-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [62b74618] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:09:32,917-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [62b74618] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:09:32,917-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [62b74618] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:09:32,917-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [62b74618] HostName = virt4 2017-10-01 16:09:32,917-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [62b74618] Command 'ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:09:32,917-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [62b74618] FINISH, ConnectStoragePoolVDSCommand, log id: 3b5bd5f7 2017-10-01 16:09:32,917-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler8) [62b74618] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:09:34,043-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [62b74618] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:09:34,043-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler8) [62b74618] Command 'SpmStatusVDSCommand(HostName = virt4, SpmStatusVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:09:34,043-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler8) [62b74618] Irs placed on server '5ae29ed2-9c0f-4754-8df3-b43d14726846' failed. Proceed Failover 2017-10-01 16:09:34,113-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [62b74618] START, ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 11ffac33 2017-10-01 16:09:34,113-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [62b74618] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:09:35,062-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [62b74618] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:09:35,062-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [62b74618] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:09:35,062-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [62b74618] HostName = virt3 2017-10-01 16:09:35,062-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [62b74618] Command 'ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:09:35,062-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [62b74618] FINISH, ConnectStoragePoolVDSCommand, log id: 11ffac33 2017-10-01 16:09:35,062-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler8) [62b74618] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:09:36,174-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler8) [62b74618] Irs placed on server 'e1e71e9d-a0cd-47db-96c6-f724f6cea736' failed. Proceed Failover 2017-10-01 16:09:36,233-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [62b74618] START, ConnectStoragePoolVDSCommand(HostName = virt1, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vdsId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 719294b9 2017-10-01 16:09:36,233-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [62b74618] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:09:37,524-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [62b74618] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt1 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:09:37,525-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [62b74618] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:09:37,525-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [62b74618] HostName = virt1 2017-10-01 16:09:37,525-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [62b74618] Command 'ConnectStoragePoolVDSCommand(HostName = virt1, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vdsId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:09:37,525-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [62b74618] FINISH, ConnectStoragePoolVDSCommand, log id: 719294b9 2017-10-01 16:09:37,525-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler8) [62b74618] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:09:37,665-04 INFO [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-47) [3595bd12] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: c8653dba-63ec-4447-872c-0ffcc6af9c0f Type: Storage 2017-10-01 16:09:37,732-04 WARN [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-47) [3595bd12] skipping connect and refresh for all hosts, last master 'true', command status 'true' 2017-10-01 16:09:37,740-04 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-6-thread-14) [62b74618] Finished reconstruct for pool 'aca61eac-ed8a-484b-bbee-3645148f6405'. Clearing event queue 2017-10-01 16:09:56,436-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler7) [5f09369b] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 16:10:00,015-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [7fd0ccc9] Autorecovering 1 storage domains 2017-10-01 16:10:00,015-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler1) [7fd0ccc9] Autorecovering storage domains id: c8653dba-63ec-4447-872c-0ffcc6af9c0f 2017-10-01 16:10:00,018-04 INFO [org.ovirt.engine.core.bll.storage.connection.ConnectDomainToStorageCommand] (DefaultQuartzScheduler1) [2f1ee60a] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: c8653dba-63ec-4447-872c-0ffcc6af9c0f Type: Storage 2017-10-01 16:10:00,019-04 INFO [org.ovirt.engine.core.bll.storage.connection.ConnectDomainToStorageCommand] (DefaultQuartzScheduler1) [2f1ee60a] ConnectDomainToStorage. Before Connect all hosts to pool. Time: Sun Oct 01 16:10:00 EDT 2017 2017-10-01 16:10:00,089-04 INFO [org.ovirt.engine.core.bll.storage.connection.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-6-thread-46) [75ddd07b] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN 2017-10-01 16:10:00,092-04 INFO [org.ovirt.engine.core.bll.storage.connection.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-6-thread-25) [448d7304] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN 2017-10-01 16:10:00,092-04 INFO [org.ovirt.engine.core.bll.storage.connection.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-6-thread-26) [12ffb486] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN 2017-10-01 16:10:00,094-04 INFO [org.ovirt.engine.core.bll.storage.connection.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-6-thread-29) [6064592] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN 2017-10-01 16:10:00,107-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-26) [12ffb486] START, ConnectStorageServerVDSCommand(HostName = virt1, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='00000000-0000-0000-0000-000000000000', storageType='NFS', connectionList='[StorageServerConnections:{id='5e557659-6817-40fa-9a55-7458125404a6', connection='virtfs-fs:/nfs/data', iqn='null', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 219a0e5e 2017-10-01 16:10:00,109-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-29) [6064592] START, ConnectStorageServerVDSCommand(HostName = virt2, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='00000000-0000-0000-0000-000000000000', storageType='NFS', connectionList='[StorageServerConnections:{id='5e557659-6817-40fa-9a55-7458125404a6', connection='virtfs-fs:/nfs/data', iqn='null', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 3abc6d2c 2017-10-01 16:10:00,126-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-46) [75ddd07b] START, ConnectStorageServerVDSCommand(HostName = virt3, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='00000000-0000-0000-0000-000000000000', storageType='NFS', connectionList='[StorageServerConnections:{id='5e557659-6817-40fa-9a55-7458125404a6', connection='virtfs-fs:/nfs/data', iqn='null', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 4d516c63 2017-10-01 16:10:00,127-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-29) [6064592] FINISH, ConnectStorageServerVDSCommand, return: {5e557659-6817-40fa-9a55-7458125404a6=0}, log id: 3abc6d2c 2017-10-01 16:10:00,128-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-25) [448d7304] START, ConnectStorageServerVDSCommand(HostName = virt4, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='00000000-0000-0000-0000-000000000000', storageType='NFS', connectionList='[StorageServerConnections:{id='5e557659-6817-40fa-9a55-7458125404a6', connection='virtfs-fs:/nfs/data', iqn='null', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 4a6e5c65 2017-10-01 16:10:00,139-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-46) [75ddd07b] FINISH, ConnectStorageServerVDSCommand, return: {5e557659-6817-40fa-9a55-7458125404a6=0}, log id: 4d516c63 2017-10-01 16:10:01,140-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-25) [448d7304] FINISH, ConnectStorageServerVDSCommand, return: {5e557659-6817-40fa-9a55-7458125404a6=0}, log id: 4a6e5c65 2017-10-01 16:10:04,657-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler10) [4b3974bc] Command 'GetAllVmStatsVDSCommand(HostName = virt1, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836'})' execution failed: VDSGenericException: VDSNetworkException: Heartbeat exceeded 2017-10-01 16:10:04,657-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.PollVmStatsRefresher] (DefaultQuartzScheduler10) [4b3974bc] Failed to fetch vms info for host 'virt1' - skipping VMs monitoring. 2017-10-01 16:10:04,660-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [6bec5a09] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt1 command SpmStatusVDS failed: Heartbeat exceeded 2017-10-01 16:10:04,660-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler4) [6bec5a09] Command 'SpmStatusVDSCommand(HostName = virt1, SpmStatusVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: VDSGenericException: VDSNetworkException: Heartbeat exceeded 2017-10-01 16:10:04,660-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3e1bd4fa] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt1 command GetStatsVDS failed: Heartbeat exceeded 2017-10-01 16:10:04,661-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStatsVDSCommand] (DefaultQuartzScheduler2) [3e1bd4fa] Command 'GetStatsVDSCommand(HostName = virt1, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vds='Host[virt1,45f8b331-842e-48e7-9df8-56adddb93836]'})' execution failed: VDSGenericException: VDSNetworkException: Heartbeat exceeded 2017-10-01 16:10:04,661-04 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [3e1bd4fa] Failed getting vds stats, host='virt1'(45f8b331-842e-48e7-9df8-56adddb93836): org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Heartbeat exceeded 2017-10-01 16:10:04,661-04 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [3e1bd4fa] Failure to refresh host 'virt1' runtime info: VDSGenericException: VDSNetworkException: Heartbeat exceeded 2017-10-01 16:10:04,661-04 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler2) [3e1bd4fa] Failed to refresh VDS, network error, continuing, vds='virt1'(45f8b331-842e-48e7-9df8-56adddb93836): VDSGenericException: VDSNetworkException: Heartbeat exceeded 2017-10-01 16:10:04,661-04 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (org.ovirt.thread.pool-6-thread-10) [3e1bd4fa] Host 'virt1' is not responding. It will stay in Connecting state for a grace period of 60 seconds and after that an attempt to fence the host will be issued. 2017-10-01 16:10:04,661-04 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (org.ovirt.thread.pool-6-thread-13) [6bec5a09] Host 'virt1' is not responding. It will stay in Connecting state for a grace period of 60 seconds and after that an attempt to fence the host will be issued. 2017-10-01 16:10:04,661-04 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (org.ovirt.thread.pool-6-thread-19) [4b3974bc] Host 'virt1' is not responding. It will stay in Connecting state for a grace period of 60 seconds and after that an attempt to fence the host will be issued. 2017-10-01 16:10:04,661-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-26) [12ffb486] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt1 command ConnectStorageServerVDS failed: Heartbeat exceeded 2017-10-01 16:10:04,661-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-26) [12ffb486] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand' return value ' 2017-10-01 16:10:04,661-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-26) [12ffb486] HostName = virt1 2017-10-01 16:10:04,661-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-26) [12ffb486] Command 'ConnectStorageServerVDSCommand(HostName = virt1, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='00000000-0000-0000-0000-000000000000', storageType='NFS', connectionList='[StorageServerConnections:{id='5e557659-6817-40fa-9a55-7458125404a6', connection='virtfs-fs:/nfs/data', iqn='null', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'})' execution failed: VDSGenericException: VDSNetworkException: Heartbeat exceeded 2017-10-01 16:10:04,661-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-26) [12ffb486] FINISH, ConnectStorageServerVDSCommand, log id: 219a0e5e 2017-10-01 16:10:04,661-04 ERROR [org.ovirt.engine.core.bll.storage.connection.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-6-thread-26) [12ffb486] Command 'org.ovirt.engine.core.bll.storage.connection.ConnectStorageToVdsCommand' failed: EngineException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Heartbeat exceeded (Failed with error VDS_NETWORK_ERROR and code 5022) 2017-10-01 16:10:04,665-04 ERROR [org.ovirt.engine.core.bll.storage.connection.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-6-thread-26) [12ffb486] Transaction rolled-back for command 'org.ovirt.engine.core.bll.storage.connection.ConnectStorageToVdsCommand'. 2017-10-01 16:10:04,666-04 INFO [org.ovirt.engine.core.bll.storage.connection.ConnectDomainToStorageCommand] (DefaultQuartzScheduler1) [2f1ee60a] ConnectDomainToStorage. After Connect all hosts to pool. Time: Sun Oct 01 16:10:04 EDT 2017 2017-10-01 16:10:04,688-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-13) [6bec5a09] EVENT_ID: VDS_HOST_NOT_RESPONDING_CONNECTING(9,008), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host virt1 is not responding. It will stay in Connecting state for a grace period of 60 seconds and after that an attempt to fence the host will be issued. 2017-10-01 16:10:04,690-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-19) [4b3974bc] EVENT_ID: VDS_HOST_NOT_RESPONDING_CONNECTING(9,008), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host virt1 is not responding. It will stay in Connecting state for a grace period of 60 seconds and after that an attempt to fence the host will be issued. 2017-10-01 16:10:04,707-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-10) [3e1bd4fa] EVENT_ID: VDS_HOST_NOT_RESPONDING_CONNECTING(9,008), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host virt1 is not responding. It will stay in Connecting state for a grace period of 60 seconds and after that an attempt to fence the host will be issued. 2017-10-01 16:10:04,750-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [6bec5a09] START, ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 263e5c32 2017-10-01 16:10:04,750-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [6bec5a09] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:10:05,870-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [6bec5a09] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:10:05,870-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [6bec5a09] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:10:05,870-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [6bec5a09] HostName = virt3 2017-10-01 16:10:05,870-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [6bec5a09] Command 'ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:10:05,870-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [6bec5a09] FINISH, ConnectStoragePoolVDSCommand, log id: 263e5c32 2017-10-01 16:10:05,870-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler4) [6bec5a09] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:10:06,361-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler4) [6bec5a09] Irs placed on server 'e1e71e9d-a0cd-47db-96c6-f724f6cea736' failed. Proceed Failover 2017-10-01 16:10:06,406-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [6bec5a09] START, ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 112cab1c 2017-10-01 16:10:06,406-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [6bec5a09] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:10:07,496-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [6bec5a09] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:10:07,496-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [6bec5a09] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:10:07,496-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [6bec5a09] HostName = virt2 2017-10-01 16:10:07,496-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [6bec5a09] Command 'ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:10:07,496-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [6bec5a09] FINISH, ConnectStoragePoolVDSCommand, log id: 112cab1c 2017-10-01 16:10:07,496-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler4) [6bec5a09] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:10:07,676-04 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to virt1-mgmt/192.168.0.34 2017-10-01 16:10:07,682-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [6bec5a09] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:10:07,682-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler4) [6bec5a09] Command 'SpmStatusVDSCommand(HostName = virt2, SpmStatusVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:10:07,682-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler4) [6bec5a09] Irs placed on server 'a0ccfd11-3402-4eba-a119-4987561b738e' failed. Proceed Failover 2017-10-01 16:10:07,727-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [6bec5a09] START, ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 369f249a 2017-10-01 16:10:07,727-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [6bec5a09] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:10:07,803-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [6bec5a09] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:10:07,803-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [6bec5a09] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:10:07,803-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [6bec5a09] HostName = virt4 2017-10-01 16:10:07,803-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [6bec5a09] Command 'ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:10:07,803-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [6bec5a09] FINISH, ConnectStoragePoolVDSCommand, log id: 369f249a 2017-10-01 16:10:07,803-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler4) [6bec5a09] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:10:07,912-04 INFO [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-15) [16abb4e9] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: c8653dba-63ec-4447-872c-0ffcc6af9c0f Type: Storage 2017-10-01 16:10:07,923-04 WARN [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-15) [16abb4e9] skipping connect and refresh for all hosts, last master 'true', command status 'true' 2017-10-01 16:10:07,932-04 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-6-thread-20) [6bec5a09] Finished reconstruct for pool 'aca61eac-ed8a-484b-bbee-3645148f6405'. Clearing event queue 2017-10-01 16:10:17,107-04 INFO [org.ovirt.engine.core.bll.storage.domain.ActivateStorageDomainCommand] (default task-121) [15159741-910d-4c5d-b8e1-fd00b498ff16] Lock Acquired to object 'EngineLock:{exclusiveLocks='[c8653dba-63ec-4447-872c-0ffcc6af9c0f=STORAGE]', sharedLocks=''}' 2017-10-01 16:10:17,169-04 INFO [org.ovirt.engine.core.bll.storage.domain.ActivateStorageDomainCommand] (org.ovirt.thread.pool-6-thread-39) [15159741-910d-4c5d-b8e1-fd00b498ff16] Running command: ActivateStorageDomainCommand internal: false. Entities affected : ID: c8653dba-63ec-4447-872c-0ffcc6af9c0f Type: StorageAction group MANIPULATE_STORAGE_DOMAIN with role type ADMIN 2017-10-01 16:10:17,175-04 INFO [org.ovirt.engine.core.bll.storage.domain.ActivateStorageDomainCommand] (org.ovirt.thread.pool-6-thread-39) [15159741-910d-4c5d-b8e1-fd00b498ff16] Lock freed to object 'EngineLock:{exclusiveLocks='[c8653dba-63ec-4447-872c-0ffcc6af9c0f=STORAGE]', sharedLocks=''}' 2017-10-01 16:10:17,176-04 INFO [org.ovirt.engine.core.bll.storage.domain.ActivateStorageDomainCommand] (org.ovirt.thread.pool-6-thread-39) [15159741-910d-4c5d-b8e1-fd00b498ff16] ActivateStorage Domain. Before Connect all hosts to pool. Time: Sun Oct 01 16:10:17 EDT 2017 2017-10-01 16:10:17,223-04 INFO [org.ovirt.engine.core.bll.storage.connection.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-6-thread-42) [43b6b758] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN 2017-10-01 16:10:17,223-04 INFO [org.ovirt.engine.core.bll.storage.connection.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-6-thread-22) [33a013cb] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN 2017-10-01 16:10:17,223-04 INFO [org.ovirt.engine.core.bll.storage.connection.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-6-thread-12) [5fdf48eb] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN 2017-10-01 16:10:17,241-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-42) [43b6b758] START, ConnectStorageServerVDSCommand(HostName = virt2, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='00000000-0000-0000-0000-000000000000', storageType='NFS', connectionList='[StorageServerConnections:{id='5e557659-6817-40fa-9a55-7458125404a6', connection='virtfs-fs:/nfs/data', iqn='null', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 44879df 2017-10-01 16:10:17,247-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-42) [43b6b758] FINISH, ConnectStorageServerVDSCommand, return: {5e557659-6817-40fa-9a55-7458125404a6=0}, log id: 44879df 2017-10-01 16:10:17,250-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-12) [5fdf48eb] START, ConnectStorageServerVDSCommand(HostName = virt4, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='00000000-0000-0000-0000-000000000000', storageType='NFS', connectionList='[StorageServerConnections:{id='5e557659-6817-40fa-9a55-7458125404a6', connection='virtfs-fs:/nfs/data', iqn='null', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 6091f042 2017-10-01 16:10:17,250-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-22) [33a013cb] START, ConnectStorageServerVDSCommand(HostName = virt3, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='00000000-0000-0000-0000-000000000000', storageType='NFS', connectionList='[StorageServerConnections:{id='5e557659-6817-40fa-9a55-7458125404a6', connection='virtfs-fs:/nfs/data', iqn='null', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 6f1365be 2017-10-01 16:10:17,256-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-12) [5fdf48eb] FINISH, ConnectStorageServerVDSCommand, return: {5e557659-6817-40fa-9a55-7458125404a6=0}, log id: 6091f042 2017-10-01 16:10:17,257-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-22) [33a013cb] FINISH, ConnectStorageServerVDSCommand, return: {5e557659-6817-40fa-9a55-7458125404a6=0}, log id: 6f1365be 2017-10-01 16:10:17,260-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (org.ovirt.thread.pool-6-thread-39) [15159741-910d-4c5d-b8e1-fd00b498ff16] START, ActivateStorageDomainVDSCommand( ActivateStorageDomainVDSCommandParameters:{runAsync='true', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', ignoreFailoverLimit='false', storageDomainId='c8653dba-63ec-4447-872c-0ffcc6af9c0f'}), log id: 24b72499 2017-10-01 16:10:17,332-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [15159741-910d-4c5d-b8e1-fd00b498ff16] START, ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 468c286f 2017-10-01 16:10:17,332-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [15159741-910d-4c5d-b8e1-fd00b498ff16] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:10:17,441-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-39) [15159741-910d-4c5d-b8e1-fd00b498ff16] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:10:17,441-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [15159741-910d-4c5d-b8e1-fd00b498ff16] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:10:17,441-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [15159741-910d-4c5d-b8e1-fd00b498ff16] HostName = virt3 2017-10-01 16:10:17,441-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [15159741-910d-4c5d-b8e1-fd00b498ff16] Command 'ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:10:17,441-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [15159741-910d-4c5d-b8e1-fd00b498ff16] FINISH, ConnectStoragePoolVDSCommand, log id: 468c286f 2017-10-01 16:10:17,441-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (org.ovirt.thread.pool-6-thread-39) [15159741-910d-4c5d-b8e1-fd00b498ff16] IrsBroker::Failed::ActivateStorageDomainVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:10:17,528-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-39) [15159741-910d-4c5d-b8e1-fd00b498ff16] Irs placed on server 'e1e71e9d-a0cd-47db-96c6-f724f6cea736' failed. Proceed Failover 2017-10-01 16:10:17,579-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [15159741-910d-4c5d-b8e1-fd00b498ff16] START, ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 6f75c6f4 2017-10-01 16:10:17,579-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [15159741-910d-4c5d-b8e1-fd00b498ff16] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:10:17,669-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-39) [15159741-910d-4c5d-b8e1-fd00b498ff16] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:10:17,669-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [15159741-910d-4c5d-b8e1-fd00b498ff16] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:10:17,669-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [15159741-910d-4c5d-b8e1-fd00b498ff16] HostName = virt2 2017-10-01 16:10:17,669-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [15159741-910d-4c5d-b8e1-fd00b498ff16] Command 'ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:10:17,669-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [15159741-910d-4c5d-b8e1-fd00b498ff16] FINISH, ConnectStoragePoolVDSCommand, log id: 6f75c6f4 2017-10-01 16:10:17,669-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (org.ovirt.thread.pool-6-thread-39) [15159741-910d-4c5d-b8e1-fd00b498ff16] IrsBroker::Failed::ActivateStorageDomainVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:10:17,763-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-39) [15159741-910d-4c5d-b8e1-fd00b498ff16] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:10:17,763-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (org.ovirt.thread.pool-6-thread-39) [15159741-910d-4c5d-b8e1-fd00b498ff16] Command 'SpmStatusVDSCommand(HostName = virt2, SpmStatusVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:10:17,763-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-39) [15159741-910d-4c5d-b8e1-fd00b498ff16] Irs placed on server 'a0ccfd11-3402-4eba-a119-4987561b738e' failed. Proceed Failover 2017-10-01 16:10:17,808-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [15159741-910d-4c5d-b8e1-fd00b498ff16] START, ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 4eeb9766 2017-10-01 16:10:17,808-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [15159741-910d-4c5d-b8e1-fd00b498ff16] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:10:17,885-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-39) [15159741-910d-4c5d-b8e1-fd00b498ff16] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:10:17,885-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [15159741-910d-4c5d-b8e1-fd00b498ff16] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:10:17,885-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [15159741-910d-4c5d-b8e1-fd00b498ff16] HostName = virt4 2017-10-01 16:10:17,885-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [15159741-910d-4c5d-b8e1-fd00b498ff16] Command 'ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:10:17,885-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-39) [15159741-910d-4c5d-b8e1-fd00b498ff16] FINISH, ConnectStoragePoolVDSCommand, log id: 4eeb9766 2017-10-01 16:10:17,885-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (org.ovirt.thread.pool-6-thread-39) [15159741-910d-4c5d-b8e1-fd00b498ff16] IrsBroker::Failed::ActivateStorageDomainVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:10:17,933-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (org.ovirt.thread.pool-6-thread-39) [15159741-910d-4c5d-b8e1-fd00b498ff16] FINISH, ActivateStorageDomainVDSCommand, log id: 24b72499 2017-10-01 16:10:17,933-04 ERROR [org.ovirt.engine.core.bll.storage.domain.ActivateStorageDomainCommand] (org.ovirt.thread.pool-6-thread-39) [15159741-910d-4c5d-b8e1-fd00b498ff16] Command 'org.ovirt.engine.core.bll.storage.domain.ActivateStorageDomainCommand' failed: EngineException: org.ovirt.engine.core.vdsbroker.irsbroker.IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' (Failed with error StoragePoolMasterNotFound and code 304) 2017-10-01 16:10:17,939-04 INFO [org.ovirt.engine.core.bll.storage.domain.ActivateStorageDomainCommand] (org.ovirt.thread.pool-6-thread-39) [15159741-910d-4c5d-b8e1-fd00b498ff16] Command [id=1d28ed17-6bcc-4c64-8612-0cedc40451fe]: Compensating CHANGED_STATUS_ONLY of org.ovirt.engine.core.common.businessentities.StoragePoolIsoMap; snapshot: EntityStatusSnapshot:{id='StoragePoolIsoMapId:{storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', storageId='c8653dba-63ec-4447-872c-0ffcc6af9c0f'}', status='Inactive'}. 2017-10-01 16:10:17,943-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-39) [15159741-910d-4c5d-b8e1-fd00b498ff16] EVENT_ID: USER_ACTIVATE_STORAGE_DOMAIN_FAILED(967), Correlation ID: 15159741-910d-4c5d-b8e1-fd00b498ff16, Job ID: 82702903-9b02-464d-8b35-10fdd76f548e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Failed to activate Storage Domain data (Data Center EECS-Primary) by admin@internal 2017-10-01 16:10:17,990-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [79fbff99] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:10:17,990-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler7) [79fbff99] Command 'SpmStatusVDSCommand(HostName = virt4, SpmStatusVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:10:17,995-04 INFO [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-17) [65dbe7b4] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: c8653dba-63ec-4447-872c-0ffcc6af9c0f Type: Storage 2017-10-01 16:10:17,996-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [79fbff99] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command HSMGetAllTasksStatusesVDS failed: Not SPM 2017-10-01 16:10:17,996-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler7) [79fbff99] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand' return value ' 2017-10-01 16:10:17,996-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler7) [79fbff99] HostName = virt4 2017-10-01 16:10:17,996-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler7) [79fbff99] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = virt4, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846'})' execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM 2017-10-01 16:10:18,008-04 WARN [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-17) [65dbe7b4] skipping connect and refresh for all hosts, last master 'true', command status 'true' 2017-10-01 16:10:18,015-04 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-6-thread-30) [15159741-910d-4c5d-b8e1-fd00b498ff16] Finished reconstruct for pool 'aca61eac-ed8a-484b-bbee-3645148f6405'. Clearing event queue 2017-10-01 16:10:18,102-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [79fbff99] START, ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 2687bfe2 2017-10-01 16:10:18,103-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [79fbff99] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:10:18,190-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [79fbff99] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:10:18,190-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [79fbff99] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:10:18,190-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [79fbff99] HostName = virt2 2017-10-01 16:10:18,190-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [79fbff99] Command 'ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:10:18,190-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [79fbff99] FINISH, ConnectStoragePoolVDSCommand, log id: 2687bfe2 2017-10-01 16:10:18,190-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler7) [79fbff99] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:10:18,283-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [79fbff99] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:10:18,284-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler7) [79fbff99] Command 'SpmStatusVDSCommand(HostName = virt2, SpmStatusVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:10:18,284-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler7) [79fbff99] Irs placed on server 'a0ccfd11-3402-4eba-a119-4987561b738e' failed. Proceed Failover 2017-10-01 16:10:18,329-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [79fbff99] START, ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 1b29206b 2017-10-01 16:10:18,329-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [79fbff99] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:10:18,433-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [79fbff99] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:10:18,433-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [79fbff99] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:10:18,433-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [79fbff99] HostName = virt3 2017-10-01 16:10:18,433-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [79fbff99] Command 'ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:10:18,433-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [79fbff99] FINISH, ConnectStoragePoolVDSCommand, log id: 1b29206b 2017-10-01 16:10:18,433-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler7) [79fbff99] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:10:18,519-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler7) [79fbff99] Irs placed on server 'e1e71e9d-a0cd-47db-96c6-f724f6cea736' failed. Proceed Failover 2017-10-01 16:10:18,563-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [79fbff99] START, ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 4ea98ea9 2017-10-01 16:10:18,563-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [79fbff99] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:10:18,636-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [79fbff99] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:10:18,636-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [79fbff99] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:10:18,636-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [79fbff99] HostName = virt4 2017-10-01 16:10:18,636-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [79fbff99] Command 'ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:10:18,636-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [79fbff99] FINISH, ConnectStoragePoolVDSCommand, log id: 4ea98ea9 2017-10-01 16:10:18,637-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler7) [79fbff99] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:10:18,737-04 INFO [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-43) [14ad6111] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: c8653dba-63ec-4447-872c-0ffcc6af9c0f Type: Storage 2017-10-01 16:10:18,748-04 WARN [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-43) [14ad6111] skipping connect and refresh for all hosts, last master 'true', command status 'true' 2017-10-01 16:10:18,755-04 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-6-thread-37) [79fbff99] Finished reconstruct for pool 'aca61eac-ed8a-484b-bbee-3645148f6405'. Clearing event queue 2017-10-01 16:10:25,721-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler5) [54bf743f] Command 'GetCapabilitiesVDSCommand(HostName = virt1, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vds='Host[virt1,45f8b331-842e-48e7-9df8-56adddb93836]'})' execution failed: java.net.NoRouteToHostException: No route to host 2017-10-01 16:10:25,724-04 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [54bf743f] Failure to refresh host 'virt1' runtime info: java.net.NoRouteToHostException: No route to host 2017-10-01 16:10:25,724-04 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler5) [54bf743f] Failed to refresh VDS, network error, continuing, vds='virt1'(45f8b331-842e-48e7-9df8-56adddb93836): java.net.NoRouteToHostException: No route to host 2017-10-01 16:10:28,738-04 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to virt1-mgmt/192.168.0.34 2017-10-01 16:10:28,742-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [54045536] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:10:28,742-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler6) [54045536] Command 'SpmStatusVDSCommand(HostName = virt4, SpmStatusVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:10:28,755-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [54045536] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command HSMGetAllTasksStatusesVDS failed: Not SPM 2017-10-01 16:10:28,755-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler6) [54045536] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand' return value ' 2017-10-01 16:10:28,756-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler6) [54045536] HostName = virt4 2017-10-01 16:10:28,756-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler6) [54045536] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = virt4, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846'})' execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM 2017-10-01 16:10:28,804-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [54045536] START, ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 6f1c418d 2017-10-01 16:10:28,805-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [54045536] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:10:28,892-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [54045536] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:10:28,892-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [54045536] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:10:28,892-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [54045536] HostName = virt2 2017-10-01 16:10:28,892-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [54045536] Command 'ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:10:28,892-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [54045536] FINISH, ConnectStoragePoolVDSCommand, log id: 6f1c418d 2017-10-01 16:10:28,892-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler6) [54045536] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:10:28,983-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [54045536] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:10:28,984-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler6) [54045536] Command 'SpmStatusVDSCommand(HostName = virt2, SpmStatusVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:10:28,984-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler6) [54045536] Irs placed on server 'a0ccfd11-3402-4eba-a119-4987561b738e' failed. Proceed Failover 2017-10-01 16:10:29,027-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [54045536] START, ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: c967552 2017-10-01 16:10:29,027-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [54045536] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:10:29,102-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [54045536] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:10:29,102-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [54045536] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:10:29,102-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [54045536] HostName = virt4 2017-10-01 16:10:29,102-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [54045536] Command 'ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:10:29,102-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [54045536] FINISH, ConnectStoragePoolVDSCommand, log id: c967552 2017-10-01 16:10:29,102-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler6) [54045536] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:10:29,195-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [54045536] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:10:29,195-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler6) [54045536] Command 'SpmStatusVDSCommand(HostName = virt4, SpmStatusVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:10:29,196-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler6) [54045536] Irs placed on server '5ae29ed2-9c0f-4754-8df3-b43d14726846' failed. Proceed Failover 2017-10-01 16:10:29,240-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [54045536] START, ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 4611595e 2017-10-01 16:10:29,240-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [54045536] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:10:29,349-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [54045536] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:10:29,349-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [54045536] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:10:29,349-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [54045536] HostName = virt3 2017-10-01 16:10:29,349-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [54045536] Command 'ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:10:29,349-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [54045536] FINISH, ConnectStoragePoolVDSCommand, log id: 4611595e 2017-10-01 16:10:29,349-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler6) [54045536] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:10:29,449-04 INFO [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-50) [4d27f401] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: c8653dba-63ec-4447-872c-0ffcc6af9c0f Type: Storage 2017-10-01 16:10:29,457-04 WARN [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-50) [4d27f401] skipping connect and refresh for all hosts, last master 'true', command status 'true' 2017-10-01 16:10:29,464-04 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-6-thread-6) [54045536] Finished reconstruct for pool 'aca61eac-ed8a-484b-bbee-3645148f6405'. Clearing event queue 2017-10-01 16:10:31,743-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler8) [7500a397] Command 'GetCapabilitiesVDSCommand(HostName = virt1, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vds='Host[virt1,45f8b331-842e-48e7-9df8-56adddb93836]'})' execution failed: java.net.NoRouteToHostException: No route to host 2017-10-01 16:10:31,743-04 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [7500a397] Failure to refresh host 'virt1' runtime info: java.net.NoRouteToHostException: No route to host 2017-10-01 16:10:31,743-04 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler8) [7500a397] Failed to refresh VDS, network error, continuing, vds='virt1'(45f8b331-842e-48e7-9df8-56adddb93836): java.net.NoRouteToHostException: No route to host 2017-10-01 16:10:34,758-04 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to virt1-mgmt/192.168.0.34 2017-10-01 16:10:37,766-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler9) [4811a61] Command 'GetCapabilitiesVDSCommand(HostName = virt1, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vds='Host[virt1,45f8b331-842e-48e7-9df8-56adddb93836]'})' execution failed: java.net.NoRouteToHostException: No route to host 2017-10-01 16:10:37,766-04 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [4811a61] Failure to refresh host 'virt1' runtime info: java.net.NoRouteToHostException: No route to host 2017-10-01 16:10:37,766-04 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler9) [4811a61] Failed to refresh VDS, network error, continuing, vds='virt1'(45f8b331-842e-48e7-9df8-56adddb93836): java.net.NoRouteToHostException: No route to host 2017-10-01 16:10:40,781-04 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to virt1-mgmt/192.168.0.34 2017-10-01 16:10:40,786-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [b5e123c] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command HSMGetAllTasksStatusesVDS failed: Not SPM: () 2017-10-01 16:10:40,786-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler3) [b5e123c] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand' return value ' 2017-10-01 16:10:40,786-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler3) [b5e123c] HostName = virt3 2017-10-01 16:10:40,786-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler3) [b5e123c] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = virt3, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736'})' execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () 2017-10-01 16:10:40,834-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [b5e123c] START, ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 1f910be5 2017-10-01 16:10:40,834-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [b5e123c] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:10:40,941-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [b5e123c] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:10:40,941-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [b5e123c] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:10:40,941-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [b5e123c] HostName = virt3 2017-10-01 16:10:40,941-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [b5e123c] Command 'ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:10:40,941-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [b5e123c] FINISH, ConnectStoragePoolVDSCommand, log id: 1f910be5 2017-10-01 16:10:40,941-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler3) [b5e123c] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:10:41,027-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler3) [b5e123c] Irs placed on server 'e1e71e9d-a0cd-47db-96c6-f724f6cea736' failed. Proceed Failover 2017-10-01 16:10:41,071-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [b5e123c] START, ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 636bdbe 2017-10-01 16:10:41,071-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [b5e123c] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:10:41,146-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [b5e123c] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:10:41,146-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [b5e123c] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:10:41,146-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [b5e123c] HostName = virt4 2017-10-01 16:10:41,146-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [b5e123c] Command 'ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:10:41,146-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [b5e123c] FINISH, ConnectStoragePoolVDSCommand, log id: 636bdbe 2017-10-01 16:10:41,146-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler3) [b5e123c] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:10:41,236-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [b5e123c] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:10:41,236-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler3) [b5e123c] Command 'SpmStatusVDSCommand(HostName = virt4, SpmStatusVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:10:41,236-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler3) [b5e123c] Irs placed on server '5ae29ed2-9c0f-4754-8df3-b43d14726846' failed. Proceed Failover 2017-10-01 16:10:41,279-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [b5e123c] START, ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 62a5fb89 2017-10-01 16:10:41,280-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [b5e123c] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:10:41,368-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [b5e123c] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:10:41,368-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [b5e123c] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:10:41,368-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [b5e123c] HostName = virt2 2017-10-01 16:10:41,368-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [b5e123c] Command 'ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:10:41,368-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [b5e123c] FINISH, ConnectStoragePoolVDSCommand, log id: 62a5fb89 2017-10-01 16:10:41,368-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler3) [b5e123c] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:10:41,471-04 INFO [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-47) [68d8ecf0] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: c8653dba-63ec-4447-872c-0ffcc6af9c0f Type: Storage 2017-10-01 16:10:41,483-04 WARN [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-47) [68d8ecf0] skipping connect and refresh for all hosts, last master 'true', command status 'true' 2017-10-01 16:10:41,497-04 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-6-thread-36) [b5e123c] Finished reconstruct for pool 'aca61eac-ed8a-484b-bbee-3645148f6405'. Clearing event queue 2017-10-01 16:10:43,787-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler8) [2477a54] Command 'GetCapabilitiesVDSCommand(HostName = virt1, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vds='Host[virt1,45f8b331-842e-48e7-9df8-56adddb93836]'})' execution failed: java.net.NoRouteToHostException: No route to host 2017-10-01 16:10:43,787-04 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [2477a54] Failure to refresh host 'virt1' runtime info: java.net.NoRouteToHostException: No route to host 2017-10-01 16:10:43,787-04 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler8) [2477a54] Failed to refresh VDS, network error, continuing, vds='virt1'(45f8b331-842e-48e7-9df8-56adddb93836): java.net.NoRouteToHostException: No route to host 2017-10-01 16:10:43,787-04 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (org.ovirt.thread.pool-6-thread-48) [2477a54] Server failed to respond, vds_id='45f8b331-842e-48e7-9df8-56adddb93836', vds_name='virt1', vm_count=0, spm_status='None', non-responsive_timeout (seconds)=60, error: java.net.NoRouteToHostException: No route to host 2017-10-01 16:10:43,811-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-48) [2477a54] EVENT_ID: VDS_FAILURE(12), Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.NoRouteToHostException: No route to host 2017-10-01 16:10:43,813-04 INFO [org.ovirt.engine.core.bll.VdsEventListener] (org.ovirt.thread.pool-6-thread-49) [2477a54] ResourceManager::vdsNotResponding entered for Host '45f8b331-842e-48e7-9df8-56adddb93836', 'virt1-mgmt' 2017-10-01 16:10:43,835-04 INFO [org.ovirt.engine.core.bll.pm.VdsNotRespondingTreatmentCommand] (org.ovirt.thread.pool-6-thread-49) [772f1dbc] Lock Acquired to object 'EngineLock:{exclusiveLocks='[45f8b331-842e-48e7-9df8-56adddb93836=VDS_FENCE]', sharedLocks=''}' 2017-10-01 16:10:43,862-04 INFO [org.ovirt.engine.core.bll.pm.VdsNotRespondingTreatmentCommand] (org.ovirt.thread.pool-6-thread-49) [772f1dbc] Running command: VdsNotRespondingTreatmentCommand internal: true. Entities affected : ID: 45f8b331-842e-48e7-9df8-56adddb93836 Type: VDS 2017-10-01 16:10:43,887-04 INFO [org.ovirt.engine.core.bll.pm.SshSoftFencingCommand] (org.ovirt.thread.pool-6-thread-49) [772f1dbc] Running command: SshSoftFencingCommand internal: true. Entities affected : ID: 45f8b331-842e-48e7-9df8-56adddb93836 Type: VDS 2017-10-01 16:10:44,005-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-49) [772f1dbc] EVENT_ID: FENCE_OPERATION_USING_AGENT_AND_PROXY_STARTED(9,020), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Executing power management status on Host virt1 using Proxy Host virt2 and Fence Agent ipmilan:virt1-idrac. 2017-10-01 16:10:44,005-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-6-thread-49) [772f1dbc] START, FenceVdsVDSCommand(HostName = virt2, FenceVdsVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', targetVdsId='45f8b331-842e-48e7-9df8-56adddb93836', action='STATUS', agent='FenceAgent:{id='b92fdde5-0407-4d57-8d00-108a0cea8300', hostId='45f8b331-842e-48e7-9df8-56adddb93836', order='1', type='ipmilan', ip='virt1-idrac', port='null', user='root', password='***', encryptOptions='false', options='privlvl=OPERATOR, delay=10, lanplus=1'}', policy='null'}), log id: 6af69bbf 2017-10-01 16:10:45,171-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-6-thread-49) [772f1dbc] FINISH, FenceVdsVDSCommand, return: FenceOperationResult:{status='SUCCESS', powerStatus='ON', message=''}, log id: 6af69bbf 2017-10-01 16:10:45,171-04 INFO [org.ovirt.engine.core.bll.pm.SshSoftFencingCommand] (org.ovirt.thread.pool-6-thread-49) [772f1dbc] Opening SSH Soft Fencing session on host 'virt1-mgmt' 2017-10-01 16:10:46,795-04 ERROR [org.ovirt.engine.core.bll.pm.SshSoftFencingCommand] (org.ovirt.thread.pool-6-thread-49) [772f1dbc] SSH Soft Fencing command failed on host 'virt1-mgmt': Failed to get the session. 2017-10-01 16:10:46,799-04 INFO [org.ovirt.engine.core.bll.pm.SshSoftFencingCommand] (org.ovirt.thread.pool-6-thread-49) [772f1dbc] Lock freed to object 'EngineLock:{exclusiveLocks='[45f8b331-842e-48e7-9df8-56adddb93836=VDS_FENCE]', sharedLocks=''}' 2017-10-01 16:10:46,804-04 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to virt1-mgmt/192.168.0.34 2017-10-01 16:10:46,828-04 INFO [org.ovirt.engine.core.bll.pm.VdsKdumpDetectionCommand] (org.ovirt.thread.pool-6-thread-49) [772f1dbc] Running command: VdsKdumpDetectionCommand internal: true. Entities affected : ID: 45f8b331-842e-48e7-9df8-56adddb93836 Type: VDS 2017-10-01 16:10:49,799-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler1) [79b07f94] Command 'GetCapabilitiesVDSCommand(HostName = virt1, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vds='Host[virt1,45f8b331-842e-48e7-9df8-56adddb93836]'})' execution failed: java.net.NoRouteToHostException: No route to host 2017-10-01 16:10:49,799-04 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [79b07f94] Failure to refresh host 'virt1' runtime info: java.net.NoRouteToHostException: No route to host 2017-10-01 16:10:51,705-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [6547e6b4] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:10:51,705-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler10) [6547e6b4] Command 'SpmStatusVDSCommand(HostName = virt2, SpmStatusVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:10:52,714-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [6547e6b4] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command HSMGetAllTasksStatusesVDS failed: Not SPM 2017-10-01 16:10:52,714-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler10) [6547e6b4] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand' return value ' 2017-10-01 16:10:52,714-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler10) [6547e6b4] HostName = virt2 2017-10-01 16:10:52,714-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler10) [6547e6b4] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = virt2, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e'})' execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM 2017-10-01 16:10:52,763-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [6547e6b4] START, ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 275f222c 2017-10-01 16:10:52,763-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [6547e6b4] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:10:52,814-04 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to virt1-mgmt/192.168.0.34 2017-10-01 16:10:52,850-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [6547e6b4] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:10:52,850-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [6547e6b4] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:10:52,850-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [6547e6b4] HostName = virt2 2017-10-01 16:10:52,851-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [6547e6b4] Command 'ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:10:52,851-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [6547e6b4] FINISH, ConnectStoragePoolVDSCommand, log id: 275f222c 2017-10-01 16:10:52,851-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler10) [6547e6b4] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:10:52,942-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [6547e6b4] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:10:52,942-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler10) [6547e6b4] Command 'SpmStatusVDSCommand(HostName = virt2, SpmStatusVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:10:52,942-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler10) [6547e6b4] Irs placed on server 'a0ccfd11-3402-4eba-a119-4987561b738e' failed. Proceed Failover 2017-10-01 16:10:52,986-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [6547e6b4] START, ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 12572a33 2017-10-01 16:10:52,986-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [6547e6b4] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:10:53,091-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [6547e6b4] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:10:53,091-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [6547e6b4] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:10:53,091-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [6547e6b4] HostName = virt3 2017-10-01 16:10:53,091-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [6547e6b4] Command 'ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:10:53,091-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [6547e6b4] FINISH, ConnectStoragePoolVDSCommand, log id: 12572a33 2017-10-01 16:10:53,091-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler10) [6547e6b4] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:10:53,176-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler10) [6547e6b4] Irs placed on server 'e1e71e9d-a0cd-47db-96c6-f724f6cea736' failed. Proceed Failover 2017-10-01 16:10:53,220-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [6547e6b4] START, ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: ac71de0 2017-10-01 16:10:53,220-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [6547e6b4] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:10:53,296-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [6547e6b4] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:10:53,296-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [6547e6b4] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:10:53,296-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [6547e6b4] HostName = virt4 2017-10-01 16:10:53,296-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [6547e6b4] Command 'ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:10:53,296-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [6547e6b4] FINISH, ConnectStoragePoolVDSCommand, log id: ac71de0 2017-10-01 16:10:53,296-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler10) [6547e6b4] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:10:53,394-04 INFO [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-28) [7837db6c] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: c8653dba-63ec-4447-872c-0ffcc6af9c0f Type: Storage 2017-10-01 16:10:53,406-04 WARN [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-28) [7837db6c] skipping connect and refresh for all hosts, last master 'true', command status 'true' 2017-10-01 16:10:53,413-04 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-6-thread-9) [6547e6b4] Finished reconstruct for pool 'aca61eac-ed8a-484b-bbee-3645148f6405'. Clearing event queue 2017-10-01 16:10:55,819-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler3) [23fd20dd] Command 'GetCapabilitiesVDSCommand(HostName = virt1, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vds='Host[virt1,45f8b331-842e-48e7-9df8-56adddb93836]'})' execution failed: java.net.NoRouteToHostException: No route to host 2017-10-01 16:10:55,819-04 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [23fd20dd] Failure to refresh host 'virt1' runtime info: java.net.NoRouteToHostException: No route to host 2017-10-01 16:10:56,456-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [43df9310] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 16:10:58,834-04 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to virt1-mgmt/192.168.0.34 2017-10-01 16:11:01,839-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler7) [25cf54cd] Command 'GetCapabilitiesVDSCommand(HostName = virt1, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vds='Host[virt1,45f8b331-842e-48e7-9df8-56adddb93836]'})' execution failed: java.net.NoRouteToHostException: No route to host 2017-10-01 16:11:01,839-04 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [25cf54cd] Failure to refresh host 'virt1' runtime info: java.net.NoRouteToHostException: No route to host 2017-10-01 16:11:04,390-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [65437735] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:11:04,390-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler6) [65437735] Command 'SpmStatusVDSCommand(HostName = virt4, SpmStatusVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:11:04,655-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [65437735] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command HSMGetAllTasksStatusesVDS failed: Not SPM 2017-10-01 16:11:04,655-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler6) [65437735] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand' return value ' 2017-10-01 16:11:04,655-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler6) [65437735] HostName = virt4 2017-10-01 16:11:04,655-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler6) [65437735] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = virt4, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846'})' execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM 2017-10-01 16:11:04,703-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [65437735] START, ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 6dbdb282 2017-10-01 16:11:04,703-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [65437735] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:11:04,855-04 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to virt1-mgmt/192.168.0.34 2017-10-01 16:11:04,859-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [65437735] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:11:04,859-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [65437735] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:11:04,859-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [65437735] HostName = virt3 2017-10-01 16:11:04,859-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [65437735] Command 'ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:11:04,859-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [65437735] FINISH, ConnectStoragePoolVDSCommand, log id: 6dbdb282 2017-10-01 16:11:04,859-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler6) [65437735] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:11:04,944-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler6) [65437735] Irs placed on server 'e1e71e9d-a0cd-47db-96c6-f724f6cea736' failed. Proceed Failover 2017-10-01 16:11:04,989-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [65437735] START, ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 472f7871 2017-10-01 16:11:04,989-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [65437735] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:11:06,640-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [65437735] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:11:06,640-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [65437735] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:11:06,640-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [65437735] HostName = virt2 2017-10-01 16:11:06,640-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [65437735] Command 'ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:11:06,640-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [65437735] FINISH, ConnectStoragePoolVDSCommand, log id: 472f7871 2017-10-01 16:11:06,640-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler6) [65437735] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:11:06,732-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [65437735] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:11:06,732-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler6) [65437735] Command 'SpmStatusVDSCommand(HostName = virt2, SpmStatusVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:11:06,732-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler6) [65437735] Irs placed on server 'a0ccfd11-3402-4eba-a119-4987561b738e' failed. Proceed Failover 2017-10-01 16:11:06,776-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [65437735] START, ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 637f23a1 2017-10-01 16:11:06,776-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [65437735] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:11:06,852-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [65437735] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:11:06,852-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [65437735] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:11:06,852-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [65437735] HostName = virt4 2017-10-01 16:11:06,852-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [65437735] Command 'ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:11:06,852-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [65437735] FINISH, ConnectStoragePoolVDSCommand, log id: 637f23a1 2017-10-01 16:11:06,852-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler6) [65437735] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:11:06,949-04 INFO [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-27) [2461bcbf] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: c8653dba-63ec-4447-872c-0ffcc6af9c0f Type: Storage 2017-10-01 16:11:06,958-04 WARN [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-27) [2461bcbf] skipping connect and refresh for all hosts, last master 'true', command status 'true' 2017-10-01 16:11:06,966-04 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-6-thread-5) [65437735] Finished reconstruct for pool 'aca61eac-ed8a-484b-bbee-3645148f6405'. Clearing event queue 2017-10-01 16:11:07,861-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler3) [b1a6af5] Command 'GetCapabilitiesVDSCommand(HostName = virt1, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vds='Host[virt1,45f8b331-842e-48e7-9df8-56adddb93836]'})' execution failed: java.net.NoRouteToHostException: No route to host 2017-10-01 16:11:07,861-04 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [b1a6af5] Failure to refresh host 'virt1' runtime info: java.net.NoRouteToHostException: No route to host 2017-10-01 16:11:10,874-04 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to virt1-mgmt/192.168.0.34 2017-10-01 16:11:13,881-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler3) [2d8ebe21] Command 'GetCapabilitiesVDSCommand(HostName = virt1, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vds='Host[virt1,45f8b331-842e-48e7-9df8-56adddb93836]'})' execution failed: java.net.NoRouteToHostException: No route to host 2017-10-01 16:11:13,882-04 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [2d8ebe21] Failure to refresh host 'virt1' runtime info: java.net.NoRouteToHostException: No route to host 2017-10-01 16:11:16,871-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-49) [772f1dbc] EVENT_ID: KDUMP_FLOW_NOT_DETECTED_ON_VDS(615), Correlation ID: 772f1dbc, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Kdump flow is not in progress on host virt1. 2017-10-01 16:11:16,872-04 WARN [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (org.ovirt.thread.pool-6-thread-49) [772f1dbc] Trying to release exclusive lock which does not exist, lock key: '45f8b331-842e-48e7-9df8-56adddb93836VDS_FENCE' 2017-10-01 16:11:16,872-04 INFO [org.ovirt.engine.core.bll.pm.VdsKdumpDetectionCommand] (org.ovirt.thread.pool-6-thread-49) [772f1dbc] Lock freed to object 'EngineLock:{exclusiveLocks='[45f8b331-842e-48e7-9df8-56adddb93836=VDS_FENCE]', sharedLocks=''}' 2017-10-01 16:11:16,897-04 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to virt1-mgmt/192.168.0.34 2017-10-01 16:11:16,946-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [1a79d959] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:11:16,946-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler9) [1a79d959] Command 'SpmStatusVDSCommand(HostName = virt4, SpmStatusVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:11:16,958-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [1a79d959] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command HSMGetAllTasksStatusesVDS failed: Not SPM 2017-10-01 16:11:16,958-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler9) [1a79d959] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand' return value ' 2017-10-01 16:11:16,958-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler9) [1a79d959] HostName = virt4 2017-10-01 16:11:16,958-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler9) [1a79d959] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = virt4, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846'})' execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM 2017-10-01 16:11:17,006-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [1a79d959] START, ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: ce820c8 2017-10-01 16:11:17,006-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [1a79d959] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:11:17,013-04 INFO [org.ovirt.engine.core.bll.pm.RestartVdsCommand] (org.ovirt.thread.pool-6-thread-49) [] Running command: RestartVdsCommand internal: true. Entities affected : ID: 45f8b331-842e-48e7-9df8-56adddb93836 Type: VDSAction group MANIPULATE_HOST with role type ADMIN 2017-10-01 16:11:17,081-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [1a79d959] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:11:17,081-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [1a79d959] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:11:17,081-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [1a79d959] HostName = virt4 2017-10-01 16:11:17,081-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [1a79d959] Command 'ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:11:17,081-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [1a79d959] FINISH, ConnectStoragePoolVDSCommand, log id: ce820c8 2017-10-01 16:11:17,081-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler9) [1a79d959] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:11:17,099-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-49) [] EVENT_ID: FENCE_OPERATION_USING_AGENT_AND_PROXY_STARTED(9,020), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Executing power management status on Host virt1 using Proxy Host virt4 and Fence Agent ipmilan:virt1-idrac. 2017-10-01 16:11:17,100-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] START, FenceVdsVDSCommand(HostName = virt4, FenceVdsVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', targetVdsId='45f8b331-842e-48e7-9df8-56adddb93836', action='STATUS', agent='FenceAgent:{id='b92fdde5-0407-4d57-8d00-108a0cea8300', hostId='45f8b331-842e-48e7-9df8-56adddb93836', order='1', type='ipmilan', ip='virt1-idrac', port='null', user='root', password='***', encryptOptions='false', options='privlvl=OPERATOR, delay=10, lanplus=1'}', policy='null'}), log id: 37ed9222 2017-10-01 16:11:17,173-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [1a79d959] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:11:17,173-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler9) [1a79d959] Command 'SpmStatusVDSCommand(HostName = virt4, SpmStatusVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:11:17,173-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler9) [1a79d959] Irs placed on server '5ae29ed2-9c0f-4754-8df3-b43d14726846' failed. Proceed Failover 2017-10-01 16:11:17,216-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [1a79d959] START, ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 77b8c789 2017-10-01 16:11:17,217-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [1a79d959] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:11:17,259-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] FINISH, FenceVdsVDSCommand, return: FenceOperationResult:{status='SUCCESS', powerStatus='ON', message=''}, log id: 37ed9222 2017-10-01 16:11:17,284-04 INFO [org.ovirt.engine.core.bll.pm.StopVdsCommand] (org.ovirt.thread.pool-6-thread-49) [39cdeb90] Running command: StopVdsCommand internal: true. Entities affected : ID: 45f8b331-842e-48e7-9df8-56adddb93836 Type: VDSAction group MANIPULATE_HOST with role type ADMIN 2017-10-01 16:11:17,284-04 INFO [org.ovirt.engine.core.bll.pm.StopVdsCommand] (org.ovirt.thread.pool-6-thread-49) [39cdeb90] Power-Management: STOP of host 'virt1' initiated. 2017-10-01 16:11:17,303-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-49) [39cdeb90] EVENT_ID: FENCE_OPERATION_STARTED(9,017), Correlation ID: 39cdeb90, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Power management stop of Host virt1 initiated. 2017-10-01 16:11:17,318-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-6-thread-49) [39cdeb90] START, SetVdsStatusVDSCommand(HostName = virt1, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', status='Reboot', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 41f02911 2017-10-01 16:11:17,327-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [1a79d959] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:11:17,327-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [1a79d959] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:11:17,327-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [1a79d959] HostName = virt3 2017-10-01 16:11:17,327-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [1a79d959] Command 'ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:11:17,327-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [1a79d959] FINISH, ConnectStoragePoolVDSCommand, log id: 77b8c789 2017-10-01 16:11:17,328-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler9) [1a79d959] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:11:17,426-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler9) [1a79d959] Irs placed on server 'e1e71e9d-a0cd-47db-96c6-f724f6cea736' failed. Proceed Failover 2017-10-01 16:11:17,471-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [1a79d959] START, ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 2d2ad23f 2017-10-01 16:11:17,471-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [1a79d959] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:11:17,561-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [1a79d959] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:11:17,561-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [1a79d959] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:11:17,561-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [1a79d959] HostName = virt2 2017-10-01 16:11:17,561-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [1a79d959] Command 'ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:11:17,561-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [1a79d959] FINISH, ConnectStoragePoolVDSCommand, log id: 2d2ad23f 2017-10-01 16:11:17,561-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler9) [1a79d959] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:11:17,661-04 INFO [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-12) [5c9ffdd2] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: c8653dba-63ec-4447-872c-0ffcc6af9c0f Type: Storage 2017-10-01 16:11:17,674-04 WARN [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-12) [5c9ffdd2] skipping connect and refresh for all hosts, last master 'true', command status 'true' 2017-10-01 16:11:17,691-04 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-6-thread-42) [1a79d959] Finished reconstruct for pool 'aca61eac-ed8a-484b-bbee-3645148f6405'. Clearing event queue 2017-10-01 16:11:19,903-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler10) [3f5b517b] Command 'GetCapabilitiesVDSCommand(HostName = virt1, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vds='Host[virt1,45f8b331-842e-48e7-9df8-56adddb93836]'})' execution failed: java.net.NoRouteToHostException: No route to host 2017-10-01 16:11:19,903-04 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [3f5b517b] Failure to refresh host 'virt1' runtime info: java.net.NoRouteToHostException: No route to host 2017-10-01 16:11:19,907-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-6-thread-49) [39cdeb90] FINISH, SetVdsStatusVDSCommand, log id: 41f02911 2017-10-01 16:11:20,027-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-49) [39cdeb90] EVENT_ID: FENCE_OPERATION_USING_AGENT_AND_PROXY_STARTED(9,020), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Executing power management stop on Host virt1 using Proxy Host virt2 and Fence Agent ipmilan:virt1-idrac. 2017-10-01 16:11:20,028-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-6-thread-49) [39cdeb90] START, FenceVdsVDSCommand(HostName = virt2, FenceVdsVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', targetVdsId='45f8b331-842e-48e7-9df8-56adddb93836', action='STOP', agent='FenceAgent:{id='b92fdde5-0407-4d57-8d00-108a0cea8300', hostId='45f8b331-842e-48e7-9df8-56adddb93836', order='1', type='ipmilan', ip='virt1-idrac', port='null', user='root', password='***', encryptOptions='false', options='privlvl=OPERATOR, delay=10, lanplus=1'}', policy='[]'}), log id: 16504c8a 2017-10-01 16:11:28,658-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [3c11e13d] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:11:28,658-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler9) [3c11e13d] Command 'SpmStatusVDSCommand(HostName = virt2, SpmStatusVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:11:28,843-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [3c11e13d] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command HSMGetAllTasksStatusesVDS failed: Not SPM 2017-10-01 16:11:28,843-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler9) [3c11e13d] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand' return value ' 2017-10-01 16:11:28,843-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler9) [3c11e13d] HostName = virt2 2017-10-01 16:11:28,843-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler9) [3c11e13d] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = virt2, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e'})' execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM 2017-10-01 16:11:28,892-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [3c11e13d] START, ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 1299cf27 2017-10-01 16:11:28,892-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [3c11e13d] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:11:33,457-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [3c11e13d] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:11:33,457-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [3c11e13d] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:11:33,457-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [3c11e13d] HostName = virt4 2017-10-01 16:11:33,457-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [3c11e13d] Command 'ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:11:33,457-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [3c11e13d] FINISH, ConnectStoragePoolVDSCommand, log id: 1299cf27 2017-10-01 16:11:33,457-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler9) [3c11e13d] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:11:33,546-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-6-thread-49) [39cdeb90] FINISH, FenceVdsVDSCommand, return: FenceOperationResult:{status='SUCCESS', powerStatus='UNKNOWN', message=''}, log id: 16504c8a 2017-10-01 16:11:33,546-04 INFO [org.ovirt.engine.core.bll.pm.SingleAgentFenceActionExecutor] (org.ovirt.thread.pool-6-thread-49) [39cdeb90] Waiting for host 'virt1-mgmt' to reach status 'OFF' 2017-10-01 16:11:34,555-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [3c11e13d] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:11:34,555-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler9) [3c11e13d] Command 'SpmStatusVDSCommand(HostName = virt4, SpmStatusVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:11:34,555-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler9) [3c11e13d] Irs placed on server '5ae29ed2-9c0f-4754-8df3-b43d14726846' failed. Proceed Failover 2017-10-01 16:11:34,606-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [3c11e13d] START, ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 1c33837c 2017-10-01 16:11:34,606-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [3c11e13d] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:11:34,714-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [3c11e13d] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:11:34,714-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [3c11e13d] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:11:34,714-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [3c11e13d] HostName = virt3 2017-10-01 16:11:34,714-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [3c11e13d] Command 'ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:11:34,714-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [3c11e13d] FINISH, ConnectStoragePoolVDSCommand, log id: 1c33837c 2017-10-01 16:11:34,714-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler9) [3c11e13d] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:11:35,800-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler9) [3c11e13d] Irs placed on server 'e1e71e9d-a0cd-47db-96c6-f724f6cea736' failed. Proceed Failover 2017-10-01 16:11:35,845-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [3c11e13d] START, ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 7ab7daa 2017-10-01 16:11:35,845-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [3c11e13d] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:11:36,935-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [3c11e13d] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:11:36,935-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [3c11e13d] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:11:36,935-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [3c11e13d] HostName = virt2 2017-10-01 16:11:36,935-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [3c11e13d] Command 'ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:11:36,935-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [3c11e13d] FINISH, ConnectStoragePoolVDSCommand, log id: 7ab7daa 2017-10-01 16:11:36,935-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler9) [3c11e13d] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:11:37,041-04 INFO [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-4) [4e6959c0] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: c8653dba-63ec-4447-872c-0ffcc6af9c0f Type: Storage 2017-10-01 16:11:37,055-04 WARN [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-4) [4e6959c0] skipping connect and refresh for all hosts, last master 'true', command status 'true' 2017-10-01 16:11:37,066-04 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-6-thread-3) [3c11e13d] Finished reconstruct for pool 'aca61eac-ed8a-484b-bbee-3645148f6405'. Clearing event queue 2017-10-01 16:11:38,546-04 INFO [org.ovirt.engine.core.bll.pm.SingleAgentFenceActionExecutor] (org.ovirt.thread.pool-6-thread-49) [39cdeb90] Attempt 1 to get host 'virt1-mgmt' status 2017-10-01 16:11:38,664-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-49) [39cdeb90] EVENT_ID: FENCE_OPERATION_USING_AGENT_AND_PROXY_STARTED(9,020), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Executing power management status on Host virt1 using Proxy Host virt4 and Fence Agent ipmilan:virt1-idrac. 2017-10-01 16:11:38,664-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-6-thread-49) [39cdeb90] START, FenceVdsVDSCommand(HostName = virt4, FenceVdsVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', targetVdsId='45f8b331-842e-48e7-9df8-56adddb93836', action='STATUS', agent='FenceAgent:{id='b92fdde5-0407-4d57-8d00-108a0cea8300', hostId='45f8b331-842e-48e7-9df8-56adddb93836', order='1', type='ipmilan', ip='virt1-idrac', port='null', user='root', password='***', encryptOptions='false', options='privlvl=OPERATOR, delay=10, lanplus=1'}', policy='[]'}), log id: 111f6456 2017-10-01 16:11:39,825-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-6-thread-49) [39cdeb90] FINISH, FenceVdsVDSCommand, return: FenceOperationResult:{status='SUCCESS', powerStatus='OFF', message=''}, log id: 111f6456 2017-10-01 16:11:39,825-04 INFO [org.ovirt.engine.core.bll.pm.SingleAgentFenceActionExecutor] (org.ovirt.thread.pool-6-thread-49) [39cdeb90] Host 'virt1-mgmt' status is 'OFF' 2017-10-01 16:11:39,865-04 INFO [org.ovirt.engine.core.bll.pm.StopVdsCommand] (org.ovirt.thread.pool-6-thread-49) [39cdeb90] Power-Management: STOP host 'virt1' succeeded. 2017-10-01 16:11:39,868-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-49) [39cdeb90] EVENT_ID: FENCE_OPERATION_SUCCEEDED(9,018), Correlation ID: 39cdeb90, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Power management stop of Host virt1 succeeded. 2017-10-01 16:11:39,873-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-49) [39cdeb90] EVENT_ID: USER_VDS_STOP(21), Correlation ID: 39cdeb90, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host virt1 was stopped by SYSTEM. 2017-10-01 16:11:39,902-04 INFO [org.ovirt.engine.core.bll.storage.FenceVdsManualyCommand] (org.ovirt.thread.pool-6-thread-49) [4591b471] Running command: FenceVdsManualyCommand internal: true. Entities affected : ID: 45f8b331-842e-48e7-9df8-56adddb93836 Type: VDSAction group MANIPULATE_HOST with role type ADMIN 2017-10-01 16:11:39,927-04 INFO [org.ovirt.engine.core.bll.ClearNonResponsiveVdsVmsCommand] (org.ovirt.thread.pool-6-thread-49) [60778982] Running command: ClearNonResponsiveVdsVmsCommand internal: true. Entities affected : ID: 45f8b331-842e-48e7-9df8-56adddb93836 Type: VDS 2017-10-01 16:11:39,984-04 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsVMsClearedVDSCommand] (org.ovirt.thread.pool-6-thread-49) [60778982] START, UpdateVdsVMsClearedVDSCommand(HostName = virt1, UpdateVdsVMsClearedVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836'}), log id: 400b1619 2017-10-01 16:11:39,984-04 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsVMsClearedVDSCommand] (org.ovirt.thread.pool-6-thread-49) [60778982] FINISH, UpdateVdsVMsClearedVDSCommand, log id: 400b1619 2017-10-01 16:11:39,991-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-49) [60778982] EVENT_ID: USER_CLEAR_UNKNOWN_VMS(348), Correlation ID: 60778982, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: All VMs' status on Non Responsive Host virt1 were changed to 'Down' by SYSTEM 2017-10-01 16:11:40,016-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-49) [60778982] EVENT_ID: VDS_AUTO_FENCE_STATUS(539), Correlation ID: 4591b471, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Auto fence for host virt1 was started. 2017-10-01 16:11:40,134-04 INFO [org.ovirt.engine.core.bll.pm.StartVdsCommand] (org.ovirt.thread.pool-6-thread-49) [] Running command: StartVdsCommand internal: true. Entities affected : ID: 45f8b331-842e-48e7-9df8-56adddb93836 Type: VDSAction group MANIPULATE_HOST with role type ADMIN 2017-10-01 16:11:40,134-04 INFO [org.ovirt.engine.core.bll.pm.StartVdsCommand] (org.ovirt.thread.pool-6-thread-49) [] Power-Management: START of host 'virt1' initiated. 2017-10-01 16:11:40,137-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-49) [] EVENT_ID: FENCE_OPERATION_STARTED(9,017), Correlation ID: 6b5f2e63, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Power management start of Host virt1 initiated. 2017-10-01 16:11:40,217-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-49) [] EVENT_ID: FENCE_OPERATION_USING_AGENT_AND_PROXY_STARTED(9,020), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Executing power management start on Host virt1 using Proxy Host virt4 and Fence Agent ipmilan:virt1-idrac. 2017-10-01 16:11:40,217-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] START, FenceVdsVDSCommand(HostName = virt4, FenceVdsVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', targetVdsId='45f8b331-842e-48e7-9df8-56adddb93836', action='START', agent='FenceAgent:{id='b92fdde5-0407-4d57-8d00-108a0cea8300', hostId='45f8b331-842e-48e7-9df8-56adddb93836', order='1', type='ipmilan', ip='virt1-idrac', port='null', user='root', password='***', encryptOptions='false', options='privlvl=OPERATOR, delay=10, lanplus=1'}', policy='[]'}), log id: 679ffbbd 2017-10-01 16:11:45,057-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] FINISH, FenceVdsVDSCommand, return: FenceOperationResult:{status='SUCCESS', powerStatus='UNKNOWN', message=''}, log id: 679ffbbd 2017-10-01 16:11:45,057-04 INFO [org.ovirt.engine.core.bll.pm.SingleAgentFenceActionExecutor] (org.ovirt.thread.pool-6-thread-49) [] Waiting for host 'virt1-mgmt' to reach status 'ON' 2017-10-01 16:11:48,036-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [5c94c820] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:11:48,037-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler2) [5c94c820] Command 'SpmStatusVDSCommand(HostName = virt2, SpmStatusVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:11:48,087-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [5c94c820] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command HSMGetAllTasksStatusesVDS failed: Not SPM 2017-10-01 16:11:48,087-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler2) [5c94c820] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand' return value ' 2017-10-01 16:11:48,087-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler2) [5c94c820] HostName = virt2 2017-10-01 16:11:48,087-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler2) [5c94c820] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = virt2, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e'})' execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM 2017-10-01 16:11:48,141-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [5c94c820] START, ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 443821fd 2017-10-01 16:11:48,141-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [5c94c820] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:11:49,219-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [5c94c820] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:11:49,219-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [5c94c820] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:11:49,219-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [5c94c820] HostName = virt4 2017-10-01 16:11:49,219-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [5c94c820] Command 'ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:11:49,219-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [5c94c820] FINISH, ConnectStoragePoolVDSCommand, log id: 443821fd 2017-10-01 16:11:49,219-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler2) [5c94c820] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:11:49,802-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [5c94c820] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:11:49,802-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler2) [5c94c820] Command 'SpmStatusVDSCommand(HostName = virt4, SpmStatusVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:11:49,802-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler2) [5c94c820] Irs placed on server '5ae29ed2-9c0f-4754-8df3-b43d14726846' failed. Proceed Failover 2017-10-01 16:11:49,845-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [5c94c820] START, ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 60575b53 2017-10-01 16:11:49,845-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [5c94c820] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:11:50,057-04 INFO [org.ovirt.engine.core.bll.pm.SingleAgentFenceActionExecutor] (org.ovirt.thread.pool-6-thread-49) [] Attempt 1 to get host 'virt1-mgmt' status 2017-10-01 16:11:50,173-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-49) [] EVENT_ID: FENCE_OPERATION_USING_AGENT_AND_PROXY_STARTED(9,020), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Executing power management status on Host virt1 using Proxy Host virt4 and Fence Agent ipmilan:virt1-idrac. 2017-10-01 16:11:50,173-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] START, FenceVdsVDSCommand(HostName = virt4, FenceVdsVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', targetVdsId='45f8b331-842e-48e7-9df8-56adddb93836', action='STATUS', agent='FenceAgent:{id='b92fdde5-0407-4d57-8d00-108a0cea8300', hostId='45f8b331-842e-48e7-9df8-56adddb93836', order='1', type='ipmilan', ip='virt1-idrac', port='null', user='root', password='***', encryptOptions='false', options='privlvl=OPERATOR, delay=10, lanplus=1'}', policy='[]'}), log id: 19a947 2017-10-01 16:11:50,180-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [5c94c820] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:11:50,180-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [5c94c820] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:11:50,180-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [5c94c820] HostName = virt3 2017-10-01 16:11:50,180-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [5c94c820] Command 'ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:11:50,180-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [5c94c820] FINISH, ConnectStoragePoolVDSCommand, log id: 60575b53 2017-10-01 16:11:50,180-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler2) [5c94c820] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:11:50,262-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] FINISH, FenceVdsVDSCommand, return: FenceOperationResult:{status='SUCCESS', powerStatus='ON', message=''}, log id: 19a947 2017-10-01 16:11:50,262-04 INFO [org.ovirt.engine.core.bll.pm.SingleAgentFenceActionExecutor] (org.ovirt.thread.pool-6-thread-49) [] Host 'virt1-mgmt' status is 'ON' 2017-10-01 16:11:50,303-04 INFO [org.ovirt.engine.core.bll.pm.StartVdsCommand] (org.ovirt.thread.pool-6-thread-49) [] Power-Management: START host 'virt1' succeeded. 2017-10-01 16:11:50,306-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-49) [] EVENT_ID: FENCE_OPERATION_SUCCEEDED(9,018), Correlation ID: 6b5f2e63, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Power management start of Host virt1 succeeded. 2017-10-01 16:11:50,307-04 INFO [org.ovirt.engine.core.bll.pm.StartVdsCommand] (org.ovirt.thread.pool-6-thread-38) [] Waiting 300 seconds, for server to finish reboot process. 2017-10-01 16:11:50,321-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-49) [] EVENT_ID: USER_VDS_START(20), Correlation ID: 6b5f2e63, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host virt1 was started by SYSTEM. 2017-10-01 16:11:50,330-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-49) [] EVENT_ID: SYSTEM_VDS_RESTART(121), Correlation ID: 772f1dbc, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host virt1 was restarted by the engine. 2017-10-01 16:11:50,330-04 WARN [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (org.ovirt.thread.pool-6-thread-49) [] Trying to release exclusive lock which does not exist, lock key: '45f8b331-842e-48e7-9df8-56adddb93836VDS_FENCE' 2017-10-01 16:11:50,330-04 INFO [org.ovirt.engine.core.bll.pm.RestartVdsCommand] (org.ovirt.thread.pool-6-thread-49) [] Lock freed to object 'EngineLock:{exclusiveLocks='[45f8b331-842e-48e7-9df8-56adddb93836=VDS_FENCE]', sharedLocks=''}' 2017-10-01 16:11:50,341-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-49) [] EVENT_ID: VDS_RECOVER(14), Correlation ID: 772f1dbc, Job ID: e35d6133-4d46-4c70-9e02-f4b05debd197, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host virt1 is rebooting. 2017-10-01 16:11:50,346-04 WARN [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (org.ovirt.thread.pool-6-thread-49) [] Trying to release exclusive lock which does not exist, lock key: '45f8b331-842e-48e7-9df8-56adddb93836VDS_FENCE' 2017-10-01 16:11:50,346-04 INFO [org.ovirt.engine.core.bll.pm.VdsNotRespondingTreatmentCommand] (org.ovirt.thread.pool-6-thread-49) [] Lock freed to object 'EngineLock:{exclusiveLocks='[45f8b331-842e-48e7-9df8-56adddb93836=VDS_FENCE]', sharedLocks=''}' 2017-10-01 16:11:51,266-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler2) [5c94c820] Irs placed on server 'e1e71e9d-a0cd-47db-96c6-f724f6cea736' failed. Proceed Failover 2017-10-01 16:11:51,311-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [5c94c820] START, ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 3d993d89 2017-10-01 16:11:51,311-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [5c94c820] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:11:52,423-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [5c94c820] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:11:52,423-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [5c94c820] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:11:52,423-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [5c94c820] HostName = virt2 2017-10-01 16:11:52,423-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [5c94c820] Command 'ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:11:52,423-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [5c94c820] FINISH, ConnectStoragePoolVDSCommand, log id: 3d993d89 2017-10-01 16:11:52,423-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler2) [5c94c820] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:11:52,522-04 INFO [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-32) [50fef5be] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: c8653dba-63ec-4447-872c-0ffcc6af9c0f Type: Storage 2017-10-01 16:11:52,532-04 WARN [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-32) [50fef5be] skipping connect and refresh for all hosts, last master 'true', command status 'true' 2017-10-01 16:11:52,539-04 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-6-thread-18) [5c94c820] Finished reconstruct for pool 'aca61eac-ed8a-484b-bbee-3645148f6405'. Clearing event queue 2017-10-01 16:11:56,475-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [455e0f08] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 16:11:59,174-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-2) [] starting processDomainRecovery for domain 'f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2:export'. 2017-10-01 16:11:59,235-04 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-2) [] Storage domain 'f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2:export' is not visible to one or more hosts. Since the domain's type is 'ImportExport', hosts status will not be changed to non-operational 2017-10-01 16:11:59,235-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-2) [] Removing vds '[]' from the domain in maintenance cache 2017-10-01 16:11:59,235-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-2) [] Removing host(s) '[]' from hosts unseen domain report cache 2017-10-01 16:12:03,262-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [5b5a5989] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:12:03,262-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler10) [5b5a5989] Command 'SpmStatusVDSCommand(HostName = virt2, SpmStatusVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:12:04,851-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [5b5a5989] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command HSMGetAllTasksStatusesVDS failed: Not SPM 2017-10-01 16:12:04,851-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler10) [5b5a5989] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand' return value ' 2017-10-01 16:12:04,851-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler10) [5b5a5989] HostName = virt2 2017-10-01 16:12:04,851-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler10) [5b5a5989] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = virt2, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e'})' execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM 2017-10-01 16:12:04,909-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [5b5a5989] START, ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 1266698f 2017-10-01 16:12:04,909-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [5b5a5989] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:12:06,000-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [5b5a5989] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:12:06,000-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [5b5a5989] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:12:06,000-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [5b5a5989] HostName = virt2 2017-10-01 16:12:06,000-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [5b5a5989] Command 'ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:12:06,000-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [5b5a5989] FINISH, ConnectStoragePoolVDSCommand, log id: 1266698f 2017-10-01 16:12:06,000-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler10) [5b5a5989] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:12:07,093-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [5b5a5989] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:12:07,093-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler10) [5b5a5989] Command 'SpmStatusVDSCommand(HostName = virt2, SpmStatusVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:12:07,094-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler10) [5b5a5989] Irs placed on server 'a0ccfd11-3402-4eba-a119-4987561b738e' failed. Proceed Failover 2017-10-01 16:12:07,144-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [5b5a5989] START, ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 798a3bb5 2017-10-01 16:12:07,144-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [5b5a5989] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:12:08,224-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [5b5a5989] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:12:08,224-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [5b5a5989] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:12:08,224-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [5b5a5989] HostName = virt4 2017-10-01 16:12:08,224-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [5b5a5989] Command 'ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:12:08,224-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [5b5a5989] FINISH, ConnectStoragePoolVDSCommand, log id: 798a3bb5 2017-10-01 16:12:08,224-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler10) [5b5a5989] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:12:09,316-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [5b5a5989] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:12:09,316-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler10) [5b5a5989] Command 'SpmStatusVDSCommand(HostName = virt4, SpmStatusVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:12:09,316-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler10) [5b5a5989] Irs placed on server '5ae29ed2-9c0f-4754-8df3-b43d14726846' failed. Proceed Failover 2017-10-01 16:12:09,361-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [5b5a5989] START, ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 437f1389 2017-10-01 16:12:09,361-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [5b5a5989] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:12:09,851-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [5b5a5989] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:12:09,851-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [5b5a5989] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:12:09,851-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [5b5a5989] HostName = virt3 2017-10-01 16:12:09,851-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [5b5a5989] Command 'ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:12:09,851-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [5b5a5989] FINISH, ConnectStoragePoolVDSCommand, log id: 437f1389 2017-10-01 16:12:09,851-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler10) [5b5a5989] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:12:09,951-04 INFO [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-24) [7ab89379] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: c8653dba-63ec-4447-872c-0ffcc6af9c0f Type: Storage 2017-10-01 16:12:09,958-04 WARN [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-24) [7ab89379] skipping connect and refresh for all hosts, last master 'true', command status 'true' 2017-10-01 16:12:09,973-04 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-6-thread-44) [5b5a5989] Finished reconstruct for pool 'aca61eac-ed8a-484b-bbee-3645148f6405'. Clearing event queue 2017-10-01 16:12:09,974-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-8) [] starting processDomainRecovery for domain '63d680c7-4c94-4b29-a6cb-f288a128b9bc:iso'. 2017-10-01 16:12:10,029-04 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-8) [] Storage domain '63d680c7-4c94-4b29-a6cb-f288a128b9bc:iso' is not visible to one or more hosts. Since the domain's type is 'ISO', hosts status will not be changed to non-operational 2017-10-01 16:12:10,029-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-8) [] Removing vds '[]' from the domain in maintenance cache 2017-10-01 16:12:10,029-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-8) [] Removing host(s) '[]' from hosts unseen domain report cache 2017-10-01 16:12:10,866-04 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-9) [] domain 'f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2:export' in problem 'NOT_REPORTED'. vds: 'virt3' 2017-10-01 16:12:10,867-04 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-9) [] domain '63d680c7-4c94-4b29-a6cb-f288a128b9bc:iso' in problem 'NOT_REPORTED'. vds: 'virt3' 2017-10-01 16:12:21,950-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [694cb26c] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command HSMGetAllTasksStatusesVDS failed: Not SPM: () 2017-10-01 16:12:21,950-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler5) [694cb26c] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand' return value ' 2017-10-01 16:12:21,950-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler5) [694cb26c] HostName = virt3 2017-10-01 16:12:21,950-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler5) [694cb26c] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = virt3, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736'})' execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () 2017-10-01 16:12:22,000-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [694cb26c] START, ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 17f31973 2017-10-01 16:12:22,000-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [694cb26c] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:12:23,075-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [694cb26c] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:12:23,075-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [694cb26c] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:12:23,075-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [694cb26c] HostName = virt4 2017-10-01 16:12:23,075-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [694cb26c] Command 'ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:12:23,075-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [694cb26c] FINISH, ConnectStoragePoolVDSCommand, log id: 17f31973 2017-10-01 16:12:23,075-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler5) [694cb26c] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:12:24,167-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [694cb26c] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:12:24,167-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler5) [694cb26c] Command 'SpmStatusVDSCommand(HostName = virt4, SpmStatusVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:12:24,167-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler5) [694cb26c] Irs placed on server '5ae29ed2-9c0f-4754-8df3-b43d14726846' failed. Proceed Failover 2017-10-01 16:12:24,210-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [694cb26c] START, ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: c73b007 2017-10-01 16:12:24,211-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [694cb26c] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:12:25,319-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [694cb26c] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:12:25,319-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [694cb26c] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:12:25,319-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [694cb26c] HostName = virt3 2017-10-01 16:12:25,319-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [694cb26c] Command 'ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:12:25,319-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [694cb26c] FINISH, ConnectStoragePoolVDSCommand, log id: c73b007 2017-10-01 16:12:25,319-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler5) [694cb26c] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:12:25,943-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler5) [694cb26c] Irs placed on server 'e1e71e9d-a0cd-47db-96c6-f724f6cea736' failed. Proceed Failover 2017-10-01 16:12:25,987-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [694cb26c] START, ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 236d2a5c 2017-10-01 16:12:25,987-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [694cb26c] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:12:27,080-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [694cb26c] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:12:27,080-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [694cb26c] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:12:27,080-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [694cb26c] HostName = virt2 2017-10-01 16:12:27,080-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [694cb26c] Command 'ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:12:27,080-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [694cb26c] FINISH, ConnectStoragePoolVDSCommand, log id: 236d2a5c 2017-10-01 16:12:27,080-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler5) [694cb26c] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:12:27,181-04 INFO [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-19) [392fae28] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: c8653dba-63ec-4447-872c-0ffcc6af9c0f Type: Storage 2017-10-01 16:12:27,194-04 WARN [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-19) [392fae28] skipping connect and refresh for all hosts, last master 'true', command status 'true' 2017-10-01 16:12:27,201-04 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-6-thread-10) [694cb26c] Finished reconstruct for pool 'aca61eac-ed8a-484b-bbee-3645148f6405'. Clearing event queue 2017-10-01 16:12:38,177-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [62522469] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:12:38,177-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler8) [62522469] Command 'SpmStatusVDSCommand(HostName = virt2, SpmStatusVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:12:39,187-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [62522469] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command HSMGetAllTasksStatusesVDS failed: Not SPM 2017-10-01 16:12:39,187-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler8) [62522469] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand' return value ' 2017-10-01 16:12:39,187-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler8) [62522469] HostName = virt2 2017-10-01 16:12:39,187-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler8) [62522469] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = virt2, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e'})' execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM 2017-10-01 16:12:39,237-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [62522469] START, ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 50773d9c 2017-10-01 16:12:39,237-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [62522469] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:12:40,315-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [62522469] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:12:40,315-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [62522469] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:12:40,315-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [62522469] HostName = virt4 2017-10-01 16:12:40,315-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [62522469] Command 'ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:12:40,315-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [62522469] FINISH, ConnectStoragePoolVDSCommand, log id: 50773d9c 2017-10-01 16:12:40,315-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler8) [62522469] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:12:41,080-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [62522469] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:12:41,080-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler8) [62522469] Command 'SpmStatusVDSCommand(HostName = virt4, SpmStatusVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:12:41,080-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler8) [62522469] Irs placed on server '5ae29ed2-9c0f-4754-8df3-b43d14726846' failed. Proceed Failover 2017-10-01 16:12:41,125-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [62522469] START, ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 4db7d10a 2017-10-01 16:12:41,125-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [62522469] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:12:42,218-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [62522469] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:12:42,218-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [62522469] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:12:42,218-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [62522469] HostName = virt2 2017-10-01 16:12:42,218-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [62522469] Command 'ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:12:42,218-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [62522469] FINISH, ConnectStoragePoolVDSCommand, log id: 4db7d10a 2017-10-01 16:12:42,218-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler8) [62522469] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:12:43,312-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [62522469] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:12:43,312-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler8) [62522469] Command 'SpmStatusVDSCommand(HostName = virt2, SpmStatusVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:12:43,312-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler8) [62522469] Irs placed on server 'a0ccfd11-3402-4eba-a119-4987561b738e' failed. Proceed Failover 2017-10-01 16:12:43,357-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [62522469] START, ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 69f48381 2017-10-01 16:12:43,357-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [62522469] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:12:44,469-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [62522469] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:12:44,469-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [62522469] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:12:44,469-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [62522469] HostName = virt3 2017-10-01 16:12:44,469-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [62522469] Command 'ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:12:44,469-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [62522469] FINISH, ConnectStoragePoolVDSCommand, log id: 69f48381 2017-10-01 16:12:44,469-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler8) [62522469] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:12:44,583-04 INFO [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-16) [4f275fef] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: c8653dba-63ec-4447-872c-0ffcc6af9c0f Type: Storage 2017-10-01 16:12:44,595-04 WARN [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-16) [4f275fef] skipping connect and refresh for all hosts, last master 'true', command status 'true' 2017-10-01 16:12:44,603-04 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-6-thread-31) [62522469] Finished reconstruct for pool 'aca61eac-ed8a-484b-bbee-3645148f6405'. Clearing event queue 2017-10-01 16:12:56,217-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [714fd6ce] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command HSMGetAllTasksStatusesVDS failed: Not SPM: () 2017-10-01 16:12:56,217-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler2) [714fd6ce] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand' return value ' 2017-10-01 16:12:56,217-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler2) [714fd6ce] HostName = virt3 2017-10-01 16:12:56,217-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler2) [714fd6ce] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = virt3, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736'})' execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () 2017-10-01 16:12:56,265-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [714fd6ce] START, ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 6aaa4e8f 2017-10-01 16:12:56,265-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [714fd6ce] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:12:56,499-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [6113b5ea] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 16:12:57,376-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [714fd6ce] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:12:57,376-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [714fd6ce] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:12:57,376-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [714fd6ce] HostName = virt3 2017-10-01 16:12:57,376-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [714fd6ce] Command 'ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:12:57,376-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [714fd6ce] FINISH, ConnectStoragePoolVDSCommand, log id: 6aaa4e8f 2017-10-01 16:12:57,376-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler2) [714fd6ce] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:12:58,463-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler2) [714fd6ce] Irs placed on server 'e1e71e9d-a0cd-47db-96c6-f724f6cea736' failed. Proceed Failover 2017-10-01 16:12:58,507-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [714fd6ce] START, ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 2df11c02 2017-10-01 16:12:58,507-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [714fd6ce] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:12:59,584-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [714fd6ce] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:12:59,584-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [714fd6ce] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:12:59,584-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [714fd6ce] HostName = virt4 2017-10-01 16:12:59,584-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [714fd6ce] Command 'ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:12:59,584-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [714fd6ce] FINISH, ConnectStoragePoolVDSCommand, log id: 2df11c02 2017-10-01 16:12:59,584-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler2) [714fd6ce] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:13:00,676-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [714fd6ce] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:13:00,676-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler2) [714fd6ce] Command 'SpmStatusVDSCommand(HostName = virt4, SpmStatusVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:13:00,676-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler2) [714fd6ce] Irs placed on server '5ae29ed2-9c0f-4754-8df3-b43d14726846' failed. Proceed Failover 2017-10-01 16:13:00,720-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [714fd6ce] START, ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 5fdaaec5 2017-10-01 16:13:00,721-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [714fd6ce] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:13:00,964-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [714fd6ce] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:13:00,964-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [714fd6ce] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:13:00,964-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [714fd6ce] HostName = virt2 2017-10-01 16:13:00,964-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [714fd6ce] Command 'ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:13:00,964-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [714fd6ce] FINISH, ConnectStoragePoolVDSCommand, log id: 5fdaaec5 2017-10-01 16:13:00,964-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler2) [714fd6ce] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:13:01,063-04 INFO [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-45) [376e0f43] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: c8653dba-63ec-4447-872c-0ffcc6af9c0f Type: Storage 2017-10-01 16:13:01,073-04 WARN [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-45) [376e0f43] skipping connect and refresh for all hosts, last master 'true', command status 'true' 2017-10-01 16:13:01,087-04 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-6-thread-30) [714fd6ce] Finished reconstruct for pool 'aca61eac-ed8a-484b-bbee-3645148f6405'. Clearing event queue 2017-10-01 16:13:11,358-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [15c9e7bc] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:13:11,358-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler7) [15c9e7bc] Command 'SpmStatusVDSCommand(HostName = virt2, SpmStatusVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:13:11,373-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [15c9e7bc] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command HSMGetAllTasksStatusesVDS failed: Not SPM 2017-10-01 16:13:11,373-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler7) [15c9e7bc] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand' return value ' 2017-10-01 16:13:11,373-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler7) [15c9e7bc] HostName = virt2 2017-10-01 16:13:11,373-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler7) [15c9e7bc] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = virt2, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e'})' execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM 2017-10-01 16:13:11,421-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [15c9e7bc] START, ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 6846158b 2017-10-01 16:13:11,421-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [15c9e7bc] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:13:12,510-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [15c9e7bc] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:13:12,510-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [15c9e7bc] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:13:12,510-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [15c9e7bc] HostName = virt2 2017-10-01 16:13:12,510-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [15c9e7bc] Command 'ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:13:12,510-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [15c9e7bc] FINISH, ConnectStoragePoolVDSCommand, log id: 6846158b 2017-10-01 16:13:12,510-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler7) [15c9e7bc] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:13:13,604-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [15c9e7bc] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:13:13,604-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler7) [15c9e7bc] Command 'SpmStatusVDSCommand(HostName = virt2, SpmStatusVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:13:13,604-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler7) [15c9e7bc] Irs placed on server 'a0ccfd11-3402-4eba-a119-4987561b738e' failed. Proceed Failover 2017-10-01 16:13:13,648-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [15c9e7bc] START, ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 2da4ac07 2017-10-01 16:13:13,648-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [15c9e7bc] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:13:14,760-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [15c9e7bc] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:13:14,760-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [15c9e7bc] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:13:14,760-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [15c9e7bc] HostName = virt3 2017-10-01 16:13:14,760-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [15c9e7bc] Command 'ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:13:14,760-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [15c9e7bc] FINISH, ConnectStoragePoolVDSCommand, log id: 2da4ac07 2017-10-01 16:13:14,760-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler7) [15c9e7bc] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:13:15,846-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler7) [15c9e7bc] Irs placed on server 'e1e71e9d-a0cd-47db-96c6-f724f6cea736' failed. Proceed Failover 2017-10-01 16:13:15,891-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [15c9e7bc] START, ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 5d2312e9 2017-10-01 16:13:15,891-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [15c9e7bc] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:13:16,969-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [15c9e7bc] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:13:16,969-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [15c9e7bc] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:13:16,969-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [15c9e7bc] HostName = virt4 2017-10-01 16:13:16,969-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [15c9e7bc] Command 'ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:13:16,969-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [15c9e7bc] FINISH, ConnectStoragePoolVDSCommand, log id: 5d2312e9 2017-10-01 16:13:16,969-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler7) [15c9e7bc] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:13:17,069-04 INFO [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-7) [5f72c576] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: c8653dba-63ec-4447-872c-0ffcc6af9c0f Type: Storage 2017-10-01 16:13:17,081-04 WARN [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-7) [5f72c576] skipping connect and refresh for all hosts, last master 'true', command status 'true' 2017-10-01 16:13:17,095-04 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-6-thread-50) [15c9e7bc] Finished reconstruct for pool 'aca61eac-ed8a-484b-bbee-3645148f6405'. Clearing event queue 2017-10-01 16:13:19,666-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler4) [22ae2ce4] Command 'GetAllVmStatsVDSCommand(HostName = virt1, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836'})' execution failed: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues 2017-10-01 16:13:19,666-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.PollVmStatsRefresher] (DefaultQuartzScheduler4) [22ae2ce4] Failed to fetch vms info for host 'virt1' - skipping VMs monitoring. 2017-10-01 16:13:19,667-04 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (org.ovirt.thread.pool-6-thread-33) [22ae2ce4] Server failed to respond, vds_id='45f8b331-842e-48e7-9df8-56adddb93836', vds_name='virt1', vm_count=0, spm_status='None', non-responsive_timeout (seconds)=60, error: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues 2017-10-01 16:13:19,697-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-33) [22ae2ce4] EVENT_ID: VDS_FAILURE(12), Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues 2017-10-01 16:13:19,699-04 INFO [org.ovirt.engine.core.bll.VdsEventListener] (org.ovirt.thread.pool-6-thread-49) [22ae2ce4] ResourceManager::vdsNotResponding entered for Host '45f8b331-842e-48e7-9df8-56adddb93836', 'virt1-mgmt' 2017-10-01 16:13:19,709-04 INFO [org.ovirt.engine.core.bll.pm.VdsNotRespondingTreatmentCommand] (org.ovirt.thread.pool-6-thread-49) [1e9c6888] Lock Acquired to object 'EngineLock:{exclusiveLocks='[45f8b331-842e-48e7-9df8-56adddb93836=VDS_FENCE]', sharedLocks=''}' 2017-10-01 16:13:19,741-04 INFO [org.ovirt.engine.core.bll.pm.VdsNotRespondingTreatmentCommand] (org.ovirt.thread.pool-6-thread-49) [1e9c6888] Running command: VdsNotRespondingTreatmentCommand internal: true. Entities affected : ID: 45f8b331-842e-48e7-9df8-56adddb93836 Type: VDS 2017-10-01 16:13:19,762-04 INFO [org.ovirt.engine.core.bll.pm.SshSoftFencingCommand] (org.ovirt.thread.pool-6-thread-49) [1e9c6888] Running command: SshSoftFencingCommand internal: true. Entities affected : ID: 45f8b331-842e-48e7-9df8-56adddb93836 Type: VDS 2017-10-01 16:13:19,894-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-49) [1e9c6888] EVENT_ID: FENCE_OPERATION_USING_AGENT_AND_PROXY_STARTED(9,020), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Executing power management status on Host virt1 using Proxy Host virt4 and Fence Agent ipmilan:virt1-idrac. 2017-10-01 16:13:19,894-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-6-thread-49) [1e9c6888] START, FenceVdsVDSCommand(HostName = virt4, FenceVdsVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', targetVdsId='45f8b331-842e-48e7-9df8-56adddb93836', action='STATUS', agent='FenceAgent:{id='b92fdde5-0407-4d57-8d00-108a0cea8300', hostId='45f8b331-842e-48e7-9df8-56adddb93836', order='1', type='ipmilan', ip='virt1-idrac', port='null', user='root', password='***', encryptOptions='false', options='privlvl=OPERATOR, delay=10, lanplus=1'}', policy='null'}), log id: 2954a0fc 2017-10-01 16:13:20,543-04 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to virt1-mgmt/192.168.0.34 2017-10-01 16:13:20,543-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-6-thread-49) [1e9c6888] FINISH, FenceVdsVDSCommand, return: FenceOperationResult:{status='SUCCESS', powerStatus='ON', message=''}, log id: 2954a0fc 2017-10-01 16:13:20,543-04 INFO [org.ovirt.engine.core.bll.pm.SshSoftFencingCommand] (org.ovirt.thread.pool-6-thread-49) [1e9c6888] Opening SSH Soft Fencing session on host 'virt1-mgmt' 2017-10-01 16:13:23,549-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler10) [2603ec4e] Command 'GetCapabilitiesVDSCommand(HostName = virt1, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vds='Host[virt1,45f8b331-842e-48e7-9df8-56adddb93836]'})' execution failed: java.net.NoRouteToHostException: No route to host 2017-10-01 16:13:23,549-04 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [2603ec4e] Failure to refresh host 'virt1' runtime info: java.net.NoRouteToHostException: No route to host 2017-10-01 16:13:23,551-04 ERROR [org.ovirt.engine.core.bll.pm.SshSoftFencingCommand] (org.ovirt.thread.pool-6-thread-49) [1e9c6888] SSH Soft Fencing command failed on host 'virt1-mgmt': Failed to get the session. 2017-10-01 16:13:23,556-04 INFO [org.ovirt.engine.core.bll.pm.SshSoftFencingCommand] (org.ovirt.thread.pool-6-thread-49) [1e9c6888] Lock freed to object 'EngineLock:{exclusiveLocks='[45f8b331-842e-48e7-9df8-56adddb93836=VDS_FENCE]', sharedLocks=''}' 2017-10-01 16:13:23,576-04 INFO [org.ovirt.engine.core.bll.pm.VdsKdumpDetectionCommand] (org.ovirt.thread.pool-6-thread-49) [1e9c6888] Running command: VdsKdumpDetectionCommand internal: true. Entities affected : ID: 45f8b331-842e-48e7-9df8-56adddb93836 Type: VDS 2017-10-01 16:13:26,564-04 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to virt1-mgmt/192.168.0.34 2017-10-01 16:13:27,063-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [665d41df] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:13:27,063-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler10) [665d41df] Command 'SpmStatusVDSCommand(HostName = virt4, SpmStatusVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:13:27,069-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [665d41df] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command HSMGetAllTasksStatusesVDS failed: Not SPM 2017-10-01 16:13:27,069-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler10) [665d41df] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand' return value ' 2017-10-01 16:13:27,069-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler10) [665d41df] HostName = virt4 2017-10-01 16:13:27,069-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler10) [665d41df] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = virt4, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846'})' execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM 2017-10-01 16:13:27,117-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [665d41df] START, ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 3362a99f 2017-10-01 16:13:27,117-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [665d41df] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:13:27,215-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [665d41df] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:13:27,215-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [665d41df] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:13:27,215-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [665d41df] HostName = virt2 2017-10-01 16:13:27,215-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [665d41df] Command 'ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:13:27,215-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [665d41df] FINISH, ConnectStoragePoolVDSCommand, log id: 3362a99f 2017-10-01 16:13:27,215-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler10) [665d41df] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:13:27,308-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [665d41df] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:13:27,308-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler10) [665d41df] Command 'SpmStatusVDSCommand(HostName = virt2, SpmStatusVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:13:27,308-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler10) [665d41df] Irs placed on server 'a0ccfd11-3402-4eba-a119-4987561b738e' failed. Proceed Failover 2017-10-01 16:13:27,352-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [665d41df] START, ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 4ccd659f 2017-10-01 16:13:27,352-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [665d41df] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:13:27,435-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [665d41df] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:13:27,435-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [665d41df] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:13:27,435-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [665d41df] HostName = virt4 2017-10-01 16:13:27,435-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [665d41df] Command 'ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:13:27,435-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [665d41df] FINISH, ConnectStoragePoolVDSCommand, log id: 4ccd659f 2017-10-01 16:13:27,435-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler10) [665d41df] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:13:27,526-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [665d41df] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:13:27,526-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler10) [665d41df] Command 'SpmStatusVDSCommand(HostName = virt4, SpmStatusVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:13:27,526-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler10) [665d41df] Irs placed on server '5ae29ed2-9c0f-4754-8df3-b43d14726846' failed. Proceed Failover 2017-10-01 16:13:27,569-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [665d41df] START, ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 7f0e6be8 2017-10-01 16:13:27,570-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [665d41df] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:13:27,677-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [665d41df] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:13:27,677-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [665d41df] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:13:27,677-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [665d41df] HostName = virt3 2017-10-01 16:13:27,677-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [665d41df] Command 'ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:13:27,677-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [665d41df] FINISH, ConnectStoragePoolVDSCommand, log id: 7f0e6be8 2017-10-01 16:13:27,677-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler10) [665d41df] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:13:27,779-04 INFO [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-41) [b5464e7] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: c8653dba-63ec-4447-872c-0ffcc6af9c0f Type: Storage 2017-10-01 16:13:27,789-04 WARN [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-41) [b5464e7] skipping connect and refresh for all hosts, last master 'true', command status 'true' 2017-10-01 16:13:27,796-04 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-6-thread-14) [665d41df] Finished reconstruct for pool 'aca61eac-ed8a-484b-bbee-3645148f6405'. Clearing event queue 2017-10-01 16:13:29,571-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler5) [7738c15e] Command 'GetCapabilitiesVDSCommand(HostName = virt1, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vds='Host[virt1,45f8b331-842e-48e7-9df8-56adddb93836]'})' execution failed: java.net.NoRouteToHostException: No route to host 2017-10-01 16:13:29,571-04 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [7738c15e] Failure to refresh host 'virt1' runtime info: java.net.NoRouteToHostException: No route to host 2017-10-01 16:13:32,615-04 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to virt1-mgmt/192.168.0.34 2017-10-01 16:13:35,621-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler2) [55bd016a] Command 'GetCapabilitiesVDSCommand(HostName = virt1, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vds='Host[virt1,45f8b331-842e-48e7-9df8-56adddb93836]'})' execution failed: java.net.NoRouteToHostException: No route to host 2017-10-01 16:13:35,621-04 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [55bd016a] Failure to refresh host 'virt1' runtime info: java.net.NoRouteToHostException: No route to host 2017-10-01 16:13:38,636-04 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to virt1-mgmt/192.168.0.34 2017-10-01 16:13:38,642-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [44661da2] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command HSMGetAllTasksStatusesVDS failed: Not SPM: () 2017-10-01 16:13:38,642-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler4) [44661da2] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand' return value ' 2017-10-01 16:13:38,642-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler4) [44661da2] HostName = virt3 2017-10-01 16:13:38,642-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler4) [44661da2] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = virt3, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736'})' execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () 2017-10-01 16:13:38,704-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [44661da2] START, ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 356e1da2 2017-10-01 16:13:38,704-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [44661da2] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:13:38,793-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [44661da2] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:13:38,793-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [44661da2] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:13:38,793-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [44661da2] HostName = virt2 2017-10-01 16:13:38,793-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [44661da2] Command 'ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:13:38,793-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [44661da2] FINISH, ConnectStoragePoolVDSCommand, log id: 356e1da2 2017-10-01 16:13:38,793-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler4) [44661da2] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:13:38,886-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [44661da2] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:13:38,886-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler4) [44661da2] Command 'SpmStatusVDSCommand(HostName = virt2, SpmStatusVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:13:38,886-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler4) [44661da2] Irs placed on server 'a0ccfd11-3402-4eba-a119-4987561b738e' failed. Proceed Failover 2017-10-01 16:13:38,931-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [44661da2] START, ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 32969b23 2017-10-01 16:13:38,931-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [44661da2] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:13:39,006-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [44661da2] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:13:39,006-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [44661da2] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:13:39,006-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [44661da2] HostName = virt4 2017-10-01 16:13:39,006-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [44661da2] Command 'ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:13:39,006-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [44661da2] FINISH, ConnectStoragePoolVDSCommand, log id: 32969b23 2017-10-01 16:13:39,006-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler4) [44661da2] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:13:39,098-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [44661da2] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:13:39,098-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler4) [44661da2] Command 'SpmStatusVDSCommand(HostName = virt4, SpmStatusVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:13:39,098-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler4) [44661da2] Irs placed on server '5ae29ed2-9c0f-4754-8df3-b43d14726846' failed. Proceed Failover 2017-10-01 16:13:39,143-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [44661da2] START, ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 2d03b451 2017-10-01 16:13:39,143-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [44661da2] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:13:39,252-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [44661da2] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:13:39,253-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [44661da2] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:13:39,253-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [44661da2] HostName = virt3 2017-10-01 16:13:39,253-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [44661da2] Command 'ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:13:39,253-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [44661da2] FINISH, ConnectStoragePoolVDSCommand, log id: 2d03b451 2017-10-01 16:13:39,253-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler4) [44661da2] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:13:39,353-04 INFO [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-29) [7798cf2a] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: c8653dba-63ec-4447-872c-0ffcc6af9c0f Type: Storage 2017-10-01 16:13:39,365-04 WARN [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-29) [7798cf2a] skipping connect and refresh for all hosts, last master 'true', command status 'true' 2017-10-01 16:13:39,379-04 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-6-thread-25) [44661da2] Finished reconstruct for pool 'aca61eac-ed8a-484b-bbee-3645148f6405'. Clearing event queue 2017-10-01 16:13:41,643-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler2) [2f4c07a3] Command 'GetCapabilitiesVDSCommand(HostName = virt1, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vds='Host[virt1,45f8b331-842e-48e7-9df8-56adddb93836]'})' execution failed: java.net.NoRouteToHostException: No route to host 2017-10-01 16:13:41,643-04 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [2f4c07a3] Failure to refresh host 'virt1' runtime info: java.net.NoRouteToHostException: No route to host 2017-10-01 16:13:44,680-04 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to virt1-mgmt/192.168.0.34 2017-10-01 16:13:47,687-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler5) [561b0e27] Command 'GetCapabilitiesVDSCommand(HostName = virt1, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vds='Host[virt1,45f8b331-842e-48e7-9df8-56adddb93836]'})' execution failed: java.net.NoRouteToHostException: No route to host 2017-10-01 16:13:47,687-04 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [561b0e27] Failure to refresh host 'virt1' runtime info: java.net.NoRouteToHostException: No route to host 2017-10-01 16:13:50,702-04 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to virt1-mgmt/192.168.0.34 2017-10-01 16:13:50,707-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [672af303] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command HSMGetAllTasksStatusesVDS failed: Not SPM: () 2017-10-01 16:13:50,708-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler9) [672af303] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand' return value ' 2017-10-01 16:13:50,710-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler9) [672af303] HostName = virt3 2017-10-01 16:13:50,710-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler9) [672af303] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = virt3, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736'})' execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () 2017-10-01 16:13:50,765-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [672af303] START, ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 2cdc597a 2017-10-01 16:13:50,767-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [672af303] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:13:50,854-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [672af303] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:13:50,854-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [672af303] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:13:50,854-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [672af303] HostName = virt2 2017-10-01 16:13:50,854-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [672af303] Command 'ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:13:50,854-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [672af303] FINISH, ConnectStoragePoolVDSCommand, log id: 2cdc597a 2017-10-01 16:13:50,854-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler9) [672af303] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:13:50,946-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [672af303] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:13:50,946-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler9) [672af303] Command 'SpmStatusVDSCommand(HostName = virt2, SpmStatusVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:13:50,946-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler9) [672af303] Irs placed on server 'a0ccfd11-3402-4eba-a119-4987561b738e' failed. Proceed Failover 2017-10-01 16:13:50,990-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [672af303] START, ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 50b0f8ec 2017-10-01 16:13:50,990-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [672af303] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:13:51,064-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [672af303] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:13:51,064-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [672af303] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:13:51,064-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [672af303] HostName = virt4 2017-10-01 16:13:51,064-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [672af303] Command 'ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:13:51,064-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [672af303] FINISH, ConnectStoragePoolVDSCommand, log id: 50b0f8ec 2017-10-01 16:13:51,064-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler9) [672af303] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:13:51,154-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [672af303] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:13:51,154-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler9) [672af303] Command 'SpmStatusVDSCommand(HostName = virt4, SpmStatusVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:13:51,154-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler9) [672af303] Irs placed on server '5ae29ed2-9c0f-4754-8df3-b43d14726846' failed. Proceed Failover 2017-10-01 16:13:51,198-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [672af303] START, ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 229baaef 2017-10-01 16:13:51,198-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [672af303] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:13:51,304-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [672af303] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:13:51,304-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [672af303] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:13:51,304-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [672af303] HostName = virt3 2017-10-01 16:13:51,304-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [672af303] Command 'ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:13:51,304-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [672af303] FINISH, ConnectStoragePoolVDSCommand, log id: 229baaef 2017-10-01 16:13:51,304-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler9) [672af303] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:13:51,405-04 INFO [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-10) [67344a65] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: c8653dba-63ec-4447-872c-0ffcc6af9c0f Type: Storage 2017-10-01 16:13:51,417-04 WARN [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-10) [67344a65] skipping connect and refresh for all hosts, last master 'true', command status 'true' 2017-10-01 16:13:51,425-04 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-6-thread-19) [672af303] Finished reconstruct for pool 'aca61eac-ed8a-484b-bbee-3645148f6405'. Clearing event queue 2017-10-01 16:13:53,604-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-49) [1e9c6888] EVENT_ID: KDUMP_FLOW_NOT_DETECTED_ON_VDS(615), Correlation ID: 1e9c6888, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Kdump flow is not in progress on host virt1. 2017-10-01 16:13:53,604-04 WARN [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (org.ovirt.thread.pool-6-thread-49) [1e9c6888] Trying to release exclusive lock which does not exist, lock key: '45f8b331-842e-48e7-9df8-56adddb93836VDS_FENCE' 2017-10-01 16:13:53,605-04 INFO [org.ovirt.engine.core.bll.pm.VdsKdumpDetectionCommand] (org.ovirt.thread.pool-6-thread-49) [1e9c6888] Lock freed to object 'EngineLock:{exclusiveLocks='[45f8b331-842e-48e7-9df8-56adddb93836=VDS_FENCE]', sharedLocks=''}' 2017-10-01 16:13:53,707-04 INFO [org.ovirt.engine.core.bll.pm.RestartVdsCommand] (org.ovirt.thread.pool-6-thread-49) [] Running command: RestartVdsCommand internal: true. Entities affected : ID: 45f8b331-842e-48e7-9df8-56adddb93836 Type: VDSAction group MANIPULATE_HOST with role type ADMIN 2017-10-01 16:13:53,709-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler6) [349edc61] Command 'GetCapabilitiesVDSCommand(HostName = virt1, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vds='Host[virt1,45f8b331-842e-48e7-9df8-56adddb93836]'})' execution failed: java.net.NoRouteToHostException: No route to host 2017-10-01 16:13:53,709-04 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [349edc61] Failure to refresh host 'virt1' runtime info: java.net.NoRouteToHostException: No route to host 2017-10-01 16:13:53,809-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-49) [] EVENT_ID: FENCE_OPERATION_USING_AGENT_AND_PROXY_STARTED(9,020), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Executing power management status on Host virt1 using Proxy Host virt2 and Fence Agent ipmilan:virt1-idrac. 2017-10-01 16:13:53,810-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] START, FenceVdsVDSCommand(HostName = virt2, FenceVdsVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', targetVdsId='45f8b331-842e-48e7-9df8-56adddb93836', action='STATUS', agent='FenceAgent:{id='b92fdde5-0407-4d57-8d00-108a0cea8300', hostId='45f8b331-842e-48e7-9df8-56adddb93836', order='1', type='ipmilan', ip='virt1-idrac', port='null', user='root', password='***', encryptOptions='false', options='privlvl=OPERATOR, delay=10, lanplus=1'}', policy='null'}), log id: 5c6ef96d 2017-10-01 16:13:54,927-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] FINISH, FenceVdsVDSCommand, return: FenceOperationResult:{status='SUCCESS', powerStatus='ON', message=''}, log id: 5c6ef96d 2017-10-01 16:13:54,932-04 INFO [org.ovirt.engine.core.bll.pm.StopVdsCommand] (org.ovirt.thread.pool-6-thread-49) [2d2d62ca] Running command: StopVdsCommand internal: true. Entities affected : ID: 45f8b331-842e-48e7-9df8-56adddb93836 Type: VDSAction group MANIPULATE_HOST with role type ADMIN 2017-10-01 16:13:54,932-04 INFO [org.ovirt.engine.core.bll.pm.StopVdsCommand] (org.ovirt.thread.pool-6-thread-49) [2d2d62ca] Power-Management: STOP of host 'virt1' initiated. 2017-10-01 16:13:54,949-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-49) [2d2d62ca] EVENT_ID: FENCE_OPERATION_STARTED(9,017), Correlation ID: 2d2d62ca, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Power management stop of Host virt1 initiated. 2017-10-01 16:13:54,964-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-6-thread-49) [2d2d62ca] START, SetVdsStatusVDSCommand(HostName = virt1, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', status='Reboot', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 2050ab68 2017-10-01 16:13:54,967-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-6-thread-49) [2d2d62ca] FINISH, SetVdsStatusVDSCommand, log id: 2050ab68 2017-10-01 16:13:55,085-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-49) [2d2d62ca] EVENT_ID: FENCE_OPERATION_USING_AGENT_AND_PROXY_STARTED(9,020), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Executing power management stop on Host virt1 using Proxy Host virt2 and Fence Agent ipmilan:virt1-idrac. 2017-10-01 16:13:55,085-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-6-thread-49) [2d2d62ca] START, FenceVdsVDSCommand(HostName = virt2, FenceVdsVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', targetVdsId='45f8b331-842e-48e7-9df8-56adddb93836', action='STOP', agent='FenceAgent:{id='b92fdde5-0407-4d57-8d00-108a0cea8300', hostId='45f8b331-842e-48e7-9df8-56adddb93836', order='1', type='ipmilan', ip='virt1-idrac', port='null', user='root', password='***', encryptOptions='false', options='privlvl=OPERATOR, delay=10, lanplus=1'}', policy='[]'}), log id: 4fe2af46 2017-10-01 16:13:56,528-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [7fa152] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 16:14:03,400-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [7285f48f] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command HSMGetAllTasksStatusesVDS failed: Not SPM: () 2017-10-01 16:14:03,400-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler10) [7285f48f] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand' return value ' 2017-10-01 16:14:03,400-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler10) [7285f48f] HostName = virt3 2017-10-01 16:14:03,400-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler10) [7285f48f] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = virt3, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736'})' execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () 2017-10-01 16:14:03,448-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [7285f48f] START, ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 4ea4ee34 2017-10-01 16:14:03,448-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [7285f48f] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:14:03,981-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [7285f48f] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:14:03,981-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [7285f48f] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:14:03,981-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [7285f48f] HostName = virt4 2017-10-01 16:14:03,981-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [7285f48f] Command 'ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:14:03,981-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [7285f48f] FINISH, ConnectStoragePoolVDSCommand, log id: 4ea4ee34 2017-10-01 16:14:03,981-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler10) [7285f48f] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:14:04,923-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [7285f48f] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:14:04,923-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler10) [7285f48f] Command 'SpmStatusVDSCommand(HostName = virt4, SpmStatusVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:14:04,923-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler10) [7285f48f] Irs placed on server '5ae29ed2-9c0f-4754-8df3-b43d14726846' failed. Proceed Failover 2017-10-01 16:14:04,968-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [7285f48f] START, ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 207f2f54 2017-10-01 16:14:04,968-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [7285f48f] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:14:06,076-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [7285f48f] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:14:06,076-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [7285f48f] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:14:06,076-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [7285f48f] HostName = virt3 2017-10-01 16:14:06,076-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [7285f48f] Command 'ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:14:06,076-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [7285f48f] FINISH, ConnectStoragePoolVDSCommand, log id: 207f2f54 2017-10-01 16:14:06,076-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler10) [7285f48f] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:14:07,163-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler10) [7285f48f] Irs placed on server 'e1e71e9d-a0cd-47db-96c6-f724f6cea736' failed. Proceed Failover 2017-10-01 16:14:07,207-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [7285f48f] START, ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: ea09c0d 2017-10-01 16:14:07,207-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [7285f48f] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:14:07,956-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [7285f48f] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:14:07,956-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [7285f48f] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:14:07,956-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [7285f48f] HostName = virt2 2017-10-01 16:14:07,956-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [7285f48f] Command 'ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:14:07,956-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [7285f48f] FINISH, ConnectStoragePoolVDSCommand, log id: ea09c0d 2017-10-01 16:14:07,956-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler10) [7285f48f] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:14:08,054-04 INFO [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-43) [670da04] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: c8653dba-63ec-4447-872c-0ffcc6af9c0f Type: Storage 2017-10-01 16:14:08,066-04 WARN [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-43) [670da04] skipping connect and refresh for all hosts, last master 'true', command status 'true' 2017-10-01 16:14:08,073-04 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-6-thread-22) [7285f48f] Finished reconstruct for pool 'aca61eac-ed8a-484b-bbee-3645148f6405'. Clearing event queue 2017-10-01 16:14:09,512-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-6-thread-49) [2d2d62ca] FINISH, FenceVdsVDSCommand, return: FenceOperationResult:{status='SUCCESS', powerStatus='UNKNOWN', message=''}, log id: 4fe2af46 2017-10-01 16:14:09,512-04 INFO [org.ovirt.engine.core.bll.pm.SingleAgentFenceActionExecutor] (org.ovirt.thread.pool-6-thread-49) [2d2d62ca] Waiting for host 'virt1-mgmt' to reach status 'OFF' 2017-10-01 16:14:14,512-04 INFO [org.ovirt.engine.core.bll.pm.SingleAgentFenceActionExecutor] (org.ovirt.thread.pool-6-thread-49) [2d2d62ca] Attempt 1 to get host 'virt1-mgmt' status 2017-10-01 16:14:14,615-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-49) [2d2d62ca] EVENT_ID: FENCE_OPERATION_USING_AGENT_AND_PROXY_STARTED(9,020), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Executing power management status on Host virt1 using Proxy Host virt4 and Fence Agent ipmilan:virt1-idrac. 2017-10-01 16:14:14,615-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-6-thread-49) [2d2d62ca] START, FenceVdsVDSCommand(HostName = virt4, FenceVdsVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', targetVdsId='45f8b331-842e-48e7-9df8-56adddb93836', action='STATUS', agent='FenceAgent:{id='b92fdde5-0407-4d57-8d00-108a0cea8300', hostId='45f8b331-842e-48e7-9df8-56adddb93836', order='1', type='ipmilan', ip='virt1-idrac', port='null', user='root', password='***', encryptOptions='false', options='privlvl=OPERATOR, delay=10, lanplus=1'}', policy='[]'}), log id: 44ae33b4 2017-10-01 16:14:15,725-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-6-thread-49) [2d2d62ca] FINISH, FenceVdsVDSCommand, return: FenceOperationResult:{status='SUCCESS', powerStatus='OFF', message=''}, log id: 44ae33b4 2017-10-01 16:14:15,725-04 INFO [org.ovirt.engine.core.bll.pm.SingleAgentFenceActionExecutor] (org.ovirt.thread.pool-6-thread-49) [2d2d62ca] Host 'virt1-mgmt' status is 'OFF' 2017-10-01 16:14:15,765-04 INFO [org.ovirt.engine.core.bll.pm.StopVdsCommand] (org.ovirt.thread.pool-6-thread-49) [2d2d62ca] Power-Management: STOP host 'virt1' succeeded. 2017-10-01 16:14:15,768-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-49) [2d2d62ca] EVENT_ID: FENCE_OPERATION_SUCCEEDED(9,018), Correlation ID: 2d2d62ca, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Power management stop of Host virt1 succeeded. 2017-10-01 16:14:15,772-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-49) [2d2d62ca] EVENT_ID: USER_VDS_STOP(21), Correlation ID: 2d2d62ca, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host virt1 was stopped by SYSTEM. 2017-10-01 16:14:15,793-04 INFO [org.ovirt.engine.core.bll.storage.FenceVdsManualyCommand] (org.ovirt.thread.pool-6-thread-49) [22330177] Running command: FenceVdsManualyCommand internal: true. Entities affected : ID: 45f8b331-842e-48e7-9df8-56adddb93836 Type: VDSAction group MANIPULATE_HOST with role type ADMIN 2017-10-01 16:14:15,815-04 INFO [org.ovirt.engine.core.bll.ClearNonResponsiveVdsVmsCommand] (org.ovirt.thread.pool-6-thread-49) [4fac280e] Running command: ClearNonResponsiveVdsVmsCommand internal: true. Entities affected : ID: 45f8b331-842e-48e7-9df8-56adddb93836 Type: VDS 2017-10-01 16:14:15,871-04 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsVMsClearedVDSCommand] (org.ovirt.thread.pool-6-thread-49) [4fac280e] START, UpdateVdsVMsClearedVDSCommand(HostName = virt1, UpdateVdsVMsClearedVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836'}), log id: 39cad23c 2017-10-01 16:14:15,871-04 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsVMsClearedVDSCommand] (org.ovirt.thread.pool-6-thread-49) [4fac280e] FINISH, UpdateVdsVMsClearedVDSCommand, log id: 39cad23c 2017-10-01 16:14:15,876-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-49) [4fac280e] EVENT_ID: USER_CLEAR_UNKNOWN_VMS(348), Correlation ID: 4fac280e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: All VMs' status on Non Responsive Host virt1 were changed to 'Down' by SYSTEM 2017-10-01 16:14:15,899-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-49) [4fac280e] EVENT_ID: VDS_AUTO_FENCE_STATUS(539), Correlation ID: 22330177, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Auto fence for host virt1 was started. 2017-10-01 16:14:15,999-04 INFO [org.ovirt.engine.core.bll.pm.StartVdsCommand] (org.ovirt.thread.pool-6-thread-49) [] Running command: StartVdsCommand internal: true. Entities affected : ID: 45f8b331-842e-48e7-9df8-56adddb93836 Type: VDSAction group MANIPULATE_HOST with role type ADMIN 2017-10-01 16:14:15,999-04 INFO [org.ovirt.engine.core.bll.pm.StartVdsCommand] (org.ovirt.thread.pool-6-thread-49) [] Power-Management: START of host 'virt1' initiated. 2017-10-01 16:14:16,002-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-49) [] EVENT_ID: FENCE_OPERATION_STARTED(9,017), Correlation ID: 6835ddcb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Power management start of Host virt1 initiated. 2017-10-01 16:14:16,104-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-49) [] EVENT_ID: FENCE_OPERATION_USING_AGENT_AND_PROXY_STARTED(9,020), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Executing power management start on Host virt1 using Proxy Host virt4 and Fence Agent ipmilan:virt1-idrac. 2017-10-01 16:14:16,105-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] START, FenceVdsVDSCommand(HostName = virt4, FenceVdsVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', targetVdsId='45f8b331-842e-48e7-9df8-56adddb93836', action='START', agent='FenceAgent:{id='b92fdde5-0407-4d57-8d00-108a0cea8300', hostId='45f8b331-842e-48e7-9df8-56adddb93836', order='1', type='ipmilan', ip='virt1-idrac', port='null', user='root', password='***', encryptOptions='false', options='privlvl=OPERATOR, delay=10, lanplus=1'}', policy='[]'}), log id: 6829aaea 2017-10-01 16:14:19,100-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [67dcf0a8] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:14:19,100-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler3) [67dcf0a8] Command 'SpmStatusVDSCommand(HostName = virt2, SpmStatusVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:14:19,710-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [67dcf0a8] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command HSMGetAllTasksStatusesVDS failed: Not SPM 2017-10-01 16:14:19,710-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler3) [67dcf0a8] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand' return value ' 2017-10-01 16:14:19,710-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler3) [67dcf0a8] HostName = virt2 2017-10-01 16:14:19,710-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler3) [67dcf0a8] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = virt2, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e'})' execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM 2017-10-01 16:14:19,759-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [67dcf0a8] START, ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 2c981e3c 2017-10-01 16:14:19,759-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [67dcf0a8] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:14:19,759-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] FINISH, FenceVdsVDSCommand, return: FenceOperationResult:{status='SUCCESS', powerStatus='UNKNOWN', message=''}, log id: 6829aaea 2017-10-01 16:14:19,759-04 INFO [org.ovirt.engine.core.bll.pm.SingleAgentFenceActionExecutor] (org.ovirt.thread.pool-6-thread-49) [] Waiting for host 'virt1-mgmt' to reach status 'ON' 2017-10-01 16:14:19,975-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [67dcf0a8] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:14:19,975-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [67dcf0a8] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:14:19,975-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [67dcf0a8] HostName = virt2 2017-10-01 16:14:19,975-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [67dcf0a8] Command 'ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:14:19,975-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [67dcf0a8] FINISH, ConnectStoragePoolVDSCommand, log id: 2c981e3c 2017-10-01 16:14:19,975-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler3) [67dcf0a8] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:14:21,068-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [67dcf0a8] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:14:21,068-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler3) [67dcf0a8] Command 'SpmStatusVDSCommand(HostName = virt2, SpmStatusVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:14:21,068-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler3) [67dcf0a8] Irs placed on server 'a0ccfd11-3402-4eba-a119-4987561b738e' failed. Proceed Failover 2017-10-01 16:14:21,112-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [67dcf0a8] START, ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 2bc092a3 2017-10-01 16:14:21,113-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [67dcf0a8] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:14:24,759-04 INFO [org.ovirt.engine.core.bll.pm.SingleAgentFenceActionExecutor] (org.ovirt.thread.pool-6-thread-49) [] Attempt 1 to get host 'virt1-mgmt' status 2017-10-01 16:14:24,875-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-49) [] EVENT_ID: FENCE_OPERATION_USING_AGENT_AND_PROXY_STARTED(9,020), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Executing power management status on Host virt1 using Proxy Host virt4 and Fence Agent ipmilan:virt1-idrac. 2017-10-01 16:14:24,875-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] START, FenceVdsVDSCommand(HostName = virt4, FenceVdsVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', targetVdsId='45f8b331-842e-48e7-9df8-56adddb93836', action='STATUS', agent='FenceAgent:{id='b92fdde5-0407-4d57-8d00-108a0cea8300', hostId='45f8b331-842e-48e7-9df8-56adddb93836', order='1', type='ipmilan', ip='virt1-idrac', port='null', user='root', password='***', encryptOptions='false', options='privlvl=OPERATOR, delay=10, lanplus=1'}', policy='[]'}), log id: 187e8783 2017-10-01 16:14:26,036-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-6-thread-49) [] FINISH, FenceVdsVDSCommand, return: FenceOperationResult:{status='SUCCESS', powerStatus='ON', message=''}, log id: 187e8783 2017-10-01 16:14:26,036-04 INFO [org.ovirt.engine.core.bll.pm.SingleAgentFenceActionExecutor] (org.ovirt.thread.pool-6-thread-49) [] Host 'virt1-mgmt' status is 'ON' 2017-10-01 16:14:26,038-04 INFO [org.ovirt.engine.core.bll.pm.StartVdsCommand] (org.ovirt.thread.pool-6-thread-49) [] Power-Management: START host 'virt1' succeeded. 2017-10-01 16:14:26,040-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-49) [] EVENT_ID: FENCE_OPERATION_SUCCEEDED(9,018), Correlation ID: 6835ddcb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Power management start of Host virt1 succeeded. 2017-10-01 16:14:26,040-04 INFO [org.ovirt.engine.core.bll.pm.StartVdsCommand] (org.ovirt.thread.pool-6-thread-3) [] Waiting 300 seconds, for server to finish reboot process. 2017-10-01 16:14:26,044-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-49) [] EVENT_ID: USER_VDS_START(20), Correlation ID: 6835ddcb, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host virt1 was started by SYSTEM. 2017-10-01 16:14:26,047-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-49) [] EVENT_ID: SYSTEM_VDS_RESTART(121), Correlation ID: 1e9c6888, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host virt1 was restarted by the engine. 2017-10-01 16:14:26,048-04 WARN [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (org.ovirt.thread.pool-6-thread-49) [] Trying to release exclusive lock which does not exist, lock key: '45f8b331-842e-48e7-9df8-56adddb93836VDS_FENCE' 2017-10-01 16:14:26,048-04 INFO [org.ovirt.engine.core.bll.pm.RestartVdsCommand] (org.ovirt.thread.pool-6-thread-49) [] Lock freed to object 'EngineLock:{exclusiveLocks='[45f8b331-842e-48e7-9df8-56adddb93836=VDS_FENCE]', sharedLocks=''}' 2017-10-01 16:14:26,051-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-49) [] EVENT_ID: VDS_RECOVER(14), Correlation ID: 1e9c6888, Job ID: 2383b01c-8a1a-4451-9c32-844dc25b9119, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host virt1 is rebooting. 2017-10-01 16:14:26,053-04 WARN [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (org.ovirt.thread.pool-6-thread-49) [] Trying to release exclusive lock which does not exist, lock key: '45f8b331-842e-48e7-9df8-56adddb93836VDS_FENCE' 2017-10-01 16:14:26,053-04 INFO [org.ovirt.engine.core.bll.pm.VdsNotRespondingTreatmentCommand] (org.ovirt.thread.pool-6-thread-49) [] Lock freed to object 'EngineLock:{exclusiveLocks='[45f8b331-842e-48e7-9df8-56adddb93836=VDS_FENCE]', sharedLocks=''}' 2017-10-01 16:14:27,086-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [67dcf0a8] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:14:27,086-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [67dcf0a8] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:14:27,086-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [67dcf0a8] HostName = virt4 2017-10-01 16:14:27,086-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [67dcf0a8] Command 'ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:14:27,086-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [67dcf0a8] FINISH, ConnectStoragePoolVDSCommand, log id: 2bc092a3 2017-10-01 16:14:27,086-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler3) [67dcf0a8] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:14:28,185-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [67dcf0a8] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:14:28,185-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler3) [67dcf0a8] Command 'SpmStatusVDSCommand(HostName = virt4, SpmStatusVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:14:28,185-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler3) [67dcf0a8] Irs placed on server '5ae29ed2-9c0f-4754-8df3-b43d14726846' failed. Proceed Failover 2017-10-01 16:14:28,229-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [67dcf0a8] START, ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 7175e808 2017-10-01 16:14:28,229-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [67dcf0a8] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:14:29,338-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [67dcf0a8] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:14:29,338-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [67dcf0a8] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:14:29,338-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [67dcf0a8] HostName = virt3 2017-10-01 16:14:29,338-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [67dcf0a8] Command 'ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:14:29,338-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [67dcf0a8] FINISH, ConnectStoragePoolVDSCommand, log id: 7175e808 2017-10-01 16:14:29,338-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler3) [67dcf0a8] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:14:29,450-04 INFO [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-35) [47a3863f] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: c8653dba-63ec-4447-872c-0ffcc6af9c0f Type: Storage 2017-10-01 16:14:29,458-04 WARN [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-35) [47a3863f] skipping connect and refresh for all hosts, last master 'true', command status 'true' 2017-10-01 16:14:29,471-04 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-6-thread-6) [67dcf0a8] Finished reconstruct for pool 'aca61eac-ed8a-484b-bbee-3645148f6405'. Clearing event queue 2017-10-01 16:14:40,524-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3f42b7c7] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command HSMGetAllTasksStatusesVDS failed: Not SPM: () 2017-10-01 16:14:40,524-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler2) [3f42b7c7] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand' return value ' 2017-10-01 16:14:40,524-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler2) [3f42b7c7] HostName = virt3 2017-10-01 16:14:40,524-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler2) [3f42b7c7] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = virt3, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736'})' execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () 2017-10-01 16:14:40,572-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [3f42b7c7] START, ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 7fe51cd2 2017-10-01 16:14:40,572-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [3f42b7c7] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:14:41,649-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3f42b7c7] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:14:41,649-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [3f42b7c7] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:14:41,649-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [3f42b7c7] HostName = virt4 2017-10-01 16:14:41,649-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [3f42b7c7] Command 'ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:14:41,649-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [3f42b7c7] FINISH, ConnectStoragePoolVDSCommand, log id: 7fe51cd2 2017-10-01 16:14:41,649-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler2) [3f42b7c7] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:14:42,741-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3f42b7c7] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:14:42,741-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler2) [3f42b7c7] Command 'SpmStatusVDSCommand(HostName = virt4, SpmStatusVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:14:42,741-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler2) [3f42b7c7] Irs placed on server '5ae29ed2-9c0f-4754-8df3-b43d14726846' failed. Proceed Failover 2017-10-01 16:14:42,785-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [3f42b7c7] START, ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 451aef7c 2017-10-01 16:14:42,785-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [3f42b7c7] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:14:43,893-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3f42b7c7] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:14:43,893-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [3f42b7c7] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:14:43,893-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [3f42b7c7] HostName = virt3 2017-10-01 16:14:43,893-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [3f42b7c7] Command 'ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:14:43,893-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [3f42b7c7] FINISH, ConnectStoragePoolVDSCommand, log id: 451aef7c 2017-10-01 16:14:43,893-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler2) [3f42b7c7] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:14:44,979-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler2) [3f42b7c7] Irs placed on server 'e1e71e9d-a0cd-47db-96c6-f724f6cea736' failed. Proceed Failover 2017-10-01 16:14:45,023-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [3f42b7c7] START, ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 609fed14 2017-10-01 16:14:45,023-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [3f42b7c7] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:14:45,817-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3f42b7c7] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:14:45,817-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [3f42b7c7] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:14:45,817-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [3f42b7c7] HostName = virt2 2017-10-01 16:14:45,817-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [3f42b7c7] Command 'ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:14:45,817-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [3f42b7c7] FINISH, ConnectStoragePoolVDSCommand, log id: 609fed14 2017-10-01 16:14:45,817-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler2) [3f42b7c7] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:14:45,917-04 INFO [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-2) [1f96126c] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: c8653dba-63ec-4447-872c-0ffcc6af9c0f Type: Storage 2017-10-01 16:14:45,924-04 WARN [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-2) [1f96126c] skipping connect and refresh for all hosts, last master 'true', command status 'true' 2017-10-01 16:14:45,938-04 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-6-thread-47) [3f42b7c7] Finished reconstruct for pool 'aca61eac-ed8a-484b-bbee-3645148f6405'. Clearing event queue 2017-10-01 16:14:56,545-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [6986e938] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 16:14:56,913-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [5b2372e0] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:14:56,913-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler4) [5b2372e0] Command 'SpmStatusVDSCommand(HostName = virt2, SpmStatusVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:14:57,921-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [5b2372e0] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command HSMGetAllTasksStatusesVDS failed: Not SPM 2017-10-01 16:14:57,921-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler4) [5b2372e0] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand' return value ' 2017-10-01 16:14:57,921-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler4) [5b2372e0] HostName = virt2 2017-10-01 16:14:57,921-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler4) [5b2372e0] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = virt2, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e'})' execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM 2017-10-01 16:14:57,971-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [5b2372e0] START, ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 4a8fc914 2017-10-01 16:14:57,971-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [5b2372e0] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:14:59,078-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [5b2372e0] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:14:59,078-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [5b2372e0] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:14:59,078-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [5b2372e0] HostName = virt3 2017-10-01 16:14:59,078-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [5b2372e0] Command 'ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:14:59,078-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [5b2372e0] FINISH, ConnectStoragePoolVDSCommand, log id: 4a8fc914 2017-10-01 16:14:59,078-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler4) [5b2372e0] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:15:00,014-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [20cd8208] Autorecovering 1 storage domains 2017-10-01 16:15:00,014-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler6) [20cd8208] Autorecovering storage domains id: c8653dba-63ec-4447-872c-0ffcc6af9c0f 2017-10-01 16:15:00,018-04 INFO [org.ovirt.engine.core.bll.storage.connection.ConnectDomainToStorageCommand] (DefaultQuartzScheduler6) [822982e] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: c8653dba-63ec-4447-872c-0ffcc6af9c0f Type: Storage 2017-10-01 16:15:00,018-04 INFO [org.ovirt.engine.core.bll.storage.connection.ConnectDomainToStorageCommand] (DefaultQuartzScheduler6) [822982e] ConnectDomainToStorage. Before Connect all hosts to pool. Time: Sun Oct 01 16:15:00 EDT 2017 2017-10-01 16:15:00,067-04 INFO [org.ovirt.engine.core.bll.storage.connection.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-6-thread-44) [69c29768] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN 2017-10-01 16:15:00,068-04 INFO [org.ovirt.engine.core.bll.storage.connection.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-6-thread-28) [7e27c148] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN 2017-10-01 16:15:00,069-04 INFO [org.ovirt.engine.core.bll.storage.connection.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-6-thread-8) [e70c645] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN 2017-10-01 16:15:00,084-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-8) [e70c645] START, ConnectStorageServerVDSCommand(HostName = virt4, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='00000000-0000-0000-0000-000000000000', storageType='NFS', connectionList='[StorageServerConnections:{id='5e557659-6817-40fa-9a55-7458125404a6', connection='virtfs-fs:/nfs/data', iqn='null', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 491c7141 2017-10-01 16:15:00,084-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-44) [69c29768] START, ConnectStorageServerVDSCommand(HostName = virt3, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='00000000-0000-0000-0000-000000000000', storageType='NFS', connectionList='[StorageServerConnections:{id='5e557659-6817-40fa-9a55-7458125404a6', connection='virtfs-fs:/nfs/data', iqn='null', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 6f7554dd 2017-10-01 16:15:00,085-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-28) [7e27c148] START, ConnectStorageServerVDSCommand(HostName = virt2, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='00000000-0000-0000-0000-000000000000', storageType='NFS', connectionList='[StorageServerConnections:{id='5e557659-6817-40fa-9a55-7458125404a6', connection='virtfs-fs:/nfs/data', iqn='null', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 6bcac9d0 2017-10-01 16:15:00,086-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler4) [5b2372e0] Irs placed on server 'e1e71e9d-a0cd-47db-96c6-f724f6cea736' failed. Proceed Failover 2017-10-01 16:15:00,097-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-8) [e70c645] FINISH, ConnectStorageServerVDSCommand, return: {5e557659-6817-40fa-9a55-7458125404a6=0}, log id: 491c7141 2017-10-01 16:15:00,097-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-28) [7e27c148] FINISH, ConnectStorageServerVDSCommand, return: {5e557659-6817-40fa-9a55-7458125404a6=0}, log id: 6bcac9d0 2017-10-01 16:15:00,138-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [5b2372e0] START, ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 4f8cfc3e 2017-10-01 16:15:00,138-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [5b2372e0] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:15:00,138-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-44) [69c29768] FINISH, ConnectStorageServerVDSCommand, return: {5e557659-6817-40fa-9a55-7458125404a6=0}, log id: 6f7554dd 2017-10-01 16:15:00,141-04 INFO [org.ovirt.engine.core.bll.storage.connection.ConnectDomainToStorageCommand] (DefaultQuartzScheduler6) [822982e] ConnectDomainToStorage. After Connect all hosts to pool. Time: Sun Oct 01 16:15:00 EDT 2017 2017-10-01 16:15:01,224-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [5b2372e0] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:15:01,224-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [5b2372e0] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:15:01,224-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [5b2372e0] HostName = virt4 2017-10-01 16:15:01,224-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [5b2372e0] Command 'ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:15:01,224-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [5b2372e0] FINISH, ConnectStoragePoolVDSCommand, log id: 4f8cfc3e 2017-10-01 16:15:01,224-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler4) [5b2372e0] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:15:01,912-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [5b2372e0] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:15:01,913-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler4) [5b2372e0] Command 'SpmStatusVDSCommand(HostName = virt4, SpmStatusVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:15:01,913-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler4) [5b2372e0] Irs placed on server '5ae29ed2-9c0f-4754-8df3-b43d14726846' failed. Proceed Failover 2017-10-01 16:15:01,957-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [5b2372e0] START, ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 2da9ed28 2017-10-01 16:15:01,957-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [5b2372e0] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:15:03,047-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [5b2372e0] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:15:03,047-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [5b2372e0] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:15:03,047-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [5b2372e0] HostName = virt2 2017-10-01 16:15:03,047-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [5b2372e0] Command 'ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:15:03,047-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [5b2372e0] FINISH, ConnectStoragePoolVDSCommand, log id: 2da9ed28 2017-10-01 16:15:03,047-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler4) [5b2372e0] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:15:03,153-04 INFO [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-26) [1da84aeb] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: c8653dba-63ec-4447-872c-0ffcc6af9c0f Type: Storage 2017-10-01 16:15:03,163-04 WARN [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-26) [1da84aeb] skipping connect and refresh for all hosts, last master 'true', command status 'true' 2017-10-01 16:15:03,171-04 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-6-thread-25) [5b2372e0] Finished reconstruct for pool 'aca61eac-ed8a-484b-bbee-3645148f6405'. Clearing event queue 2017-10-01 16:15:14,142-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [7e621f35] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:15:14,142-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler7) [7e621f35] Command 'SpmStatusVDSCommand(HostName = virt2, SpmStatusVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:15:15,150-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [7e621f35] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command HSMGetAllTasksStatusesVDS failed: Not SPM 2017-10-01 16:15:15,150-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler7) [7e621f35] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand' return value ' 2017-10-01 16:15:15,150-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler7) [7e621f35] HostName = virt2 2017-10-01 16:15:15,150-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler7) [7e621f35] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = virt2, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e'})' execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM 2017-10-01 16:15:15,199-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [7e621f35] START, ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 396bf91d 2017-10-01 16:15:15,199-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [7e621f35] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:15:16,275-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [7e621f35] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:15:16,275-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [7e621f35] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:15:16,275-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [7e621f35] HostName = virt4 2017-10-01 16:15:16,275-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [7e621f35] Command 'ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:15:16,275-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [7e621f35] FINISH, ConnectStoragePoolVDSCommand, log id: 396bf91d 2017-10-01 16:15:16,275-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler7) [7e621f35] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:15:17,048-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [7e621f35] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:15:17,048-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler7) [7e621f35] Command 'SpmStatusVDSCommand(HostName = virt4, SpmStatusVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:15:17,048-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler7) [7e621f35] Irs placed on server '5ae29ed2-9c0f-4754-8df3-b43d14726846' failed. Proceed Failover 2017-10-01 16:15:17,092-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [7e621f35] START, ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: c592e73 2017-10-01 16:15:17,092-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [7e621f35] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:15:18,202-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [7e621f35] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:15:18,202-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [7e621f35] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:15:18,202-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [7e621f35] HostName = virt3 2017-10-01 16:15:18,202-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [7e621f35] Command 'ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:15:18,202-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [7e621f35] FINISH, ConnectStoragePoolVDSCommand, log id: c592e73 2017-10-01 16:15:18,202-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler7) [7e621f35] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:15:19,290-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler7) [7e621f35] Irs placed on server 'e1e71e9d-a0cd-47db-96c6-f724f6cea736' failed. Proceed Failover 2017-10-01 16:15:19,334-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [7e621f35] START, ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 4ade2fca 2017-10-01 16:15:19,334-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [7e621f35] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:15:20,425-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [7e621f35] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:15:20,425-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [7e621f35] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:15:20,425-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [7e621f35] HostName = virt2 2017-10-01 16:15:20,425-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [7e621f35] Command 'ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:15:20,425-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [7e621f35] FINISH, ConnectStoragePoolVDSCommand, log id: 4ade2fca 2017-10-01 16:15:20,425-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler7) [7e621f35] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:15:20,524-04 INFO [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-15) [37f1b7ad] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: c8653dba-63ec-4447-872c-0ffcc6af9c0f Type: Storage 2017-10-01 16:15:20,531-04 WARN [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-15) [37f1b7ad] skipping connect and refresh for all hosts, last master 'true', command status 'true' 2017-10-01 16:15:20,546-04 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-6-thread-5) [7e621f35] Finished reconstruct for pool 'aca61eac-ed8a-484b-bbee-3645148f6405'. Clearing event queue 2017-10-01 16:15:31,522-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [46c05762] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:15:31,522-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler3) [46c05762] Command 'SpmStatusVDSCommand(HostName = virt2, SpmStatusVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:15:32,181-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [46c05762] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command HSMGetAllTasksStatusesVDS failed: Not SPM 2017-10-01 16:15:32,181-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler3) [46c05762] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand' return value ' 2017-10-01 16:15:32,181-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler3) [46c05762] HostName = virt2 2017-10-01 16:15:32,181-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler3) [46c05762] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = virt2, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e'})' execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM 2017-10-01 16:15:32,230-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [46c05762] START, ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 3bd754f5 2017-10-01 16:15:32,230-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [46c05762] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:15:33,307-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [46c05762] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:15:33,307-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [46c05762] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:15:33,307-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [46c05762] HostName = virt4 2017-10-01 16:15:33,307-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [46c05762] Command 'ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:15:33,307-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [46c05762] FINISH, ConnectStoragePoolVDSCommand, log id: 3bd754f5 2017-10-01 16:15:33,307-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler3) [46c05762] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:15:34,399-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [46c05762] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:15:34,399-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler3) [46c05762] Command 'SpmStatusVDSCommand(HostName = virt4, SpmStatusVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:15:34,399-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler3) [46c05762] Irs placed on server '5ae29ed2-9c0f-4754-8df3-b43d14726846' failed. Proceed Failover 2017-10-01 16:15:34,442-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [46c05762] START, ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 4941dcf7 2017-10-01 16:15:34,443-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [46c05762] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:15:35,531-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [46c05762] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:15:35,531-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [46c05762] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:15:35,531-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [46c05762] HostName = virt2 2017-10-01 16:15:35,531-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [46c05762] Command 'ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:15:35,531-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [46c05762] FINISH, ConnectStoragePoolVDSCommand, log id: 4941dcf7 2017-10-01 16:15:35,531-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler3) [46c05762] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:15:36,625-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [46c05762] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:15:36,625-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler3) [46c05762] Command 'SpmStatusVDSCommand(HostName = virt2, SpmStatusVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:15:36,625-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler3) [46c05762] Irs placed on server 'a0ccfd11-3402-4eba-a119-4987561b738e' failed. Proceed Failover 2017-10-01 16:15:36,669-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [46c05762] START, ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 5d15e42c 2017-10-01 16:15:36,669-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [46c05762] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:15:39,716-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [46c05762] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:15:39,716-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [46c05762] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:15:39,716-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [46c05762] HostName = virt3 2017-10-01 16:15:39,716-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [46c05762] Command 'ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:15:39,716-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [46c05762] FINISH, ConnectStoragePoolVDSCommand, log id: 5d15e42c 2017-10-01 16:15:39,716-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler3) [46c05762] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:15:39,818-04 INFO [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-12) [5713e168] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: c8653dba-63ec-4447-872c-0ffcc6af9c0f Type: Storage 2017-10-01 16:15:39,830-04 WARN [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-12) [5713e168] skipping connect and refresh for all hosts, last master 'true', command status 'true' 2017-10-01 16:15:39,837-04 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-6-thread-17) [46c05762] Finished reconstruct for pool 'aca61eac-ed8a-484b-bbee-3645148f6405'. Clearing event queue 2017-10-01 16:15:51,814-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [60dcfc4b] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command HSMGetAllTasksStatusesVDS failed: Not SPM: () 2017-10-01 16:15:51,814-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler6) [60dcfc4b] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand' return value ' 2017-10-01 16:15:51,814-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler6) [60dcfc4b] HostName = virt3 2017-10-01 16:15:51,814-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler6) [60dcfc4b] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = virt3, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736'})' execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () 2017-10-01 16:15:51,862-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [60dcfc4b] START, ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 6cd7f472 2017-10-01 16:15:51,863-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [60dcfc4b] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:15:52,970-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [60dcfc4b] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:15:52,970-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [60dcfc4b] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:15:52,970-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [60dcfc4b] HostName = virt3 2017-10-01 16:15:52,970-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [60dcfc4b] Command 'ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:15:52,970-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [60dcfc4b] FINISH, ConnectStoragePoolVDSCommand, log id: 6cd7f472 2017-10-01 16:15:52,970-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler6) [60dcfc4b] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:15:54,056-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler6) [60dcfc4b] Irs placed on server 'e1e71e9d-a0cd-47db-96c6-f724f6cea736' failed. Proceed Failover 2017-10-01 16:15:54,101-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [60dcfc4b] START, ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 4684c062 2017-10-01 16:15:54,101-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [60dcfc4b] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:15:54,361-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [60dcfc4b] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:15:54,361-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [60dcfc4b] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:15:54,361-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [60dcfc4b] HostName = virt2 2017-10-01 16:15:54,361-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [60dcfc4b] Command 'ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:15:54,361-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [60dcfc4b] FINISH, ConnectStoragePoolVDSCommand, log id: 4684c062 2017-10-01 16:15:54,361-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler6) [60dcfc4b] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:15:55,454-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [60dcfc4b] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:15:55,454-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler6) [60dcfc4b] Command 'SpmStatusVDSCommand(HostName = virt2, SpmStatusVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:15:55,454-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler6) [60dcfc4b] Irs placed on server 'a0ccfd11-3402-4eba-a119-4987561b738e' failed. Proceed Failover 2017-10-01 16:15:55,498-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [60dcfc4b] START, ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 7a9c1744 2017-10-01 16:15:55,498-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [60dcfc4b] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:15:55,719-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [60dcfc4b] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:15:55,719-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [60dcfc4b] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:15:55,719-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [60dcfc4b] HostName = virt4 2017-10-01 16:15:55,719-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [60dcfc4b] Command 'ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:15:55,719-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [60dcfc4b] FINISH, ConnectStoragePoolVDSCommand, log id: 7a9c1744 2017-10-01 16:15:55,719-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler6) [60dcfc4b] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:15:55,818-04 INFO [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-37) [2e594baa] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: c8653dba-63ec-4447-872c-0ffcc6af9c0f Type: Storage 2017-10-01 16:15:55,830-04 WARN [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-37) [2e594baa] skipping connect and refresh for all hosts, last master 'true', command status 'true' 2017-10-01 16:15:55,837-04 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-6-thread-21) [60dcfc4b] Finished reconstruct for pool 'aca61eac-ed8a-484b-bbee-3645148f6405'. Clearing event queue 2017-10-01 16:15:56,565-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [450f6a7c] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 16:16:06,822-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [1ca32bd1] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:16:06,822-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler6) [1ca32bd1] Command 'SpmStatusVDSCommand(HostName = virt4, SpmStatusVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:16:07,831-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [1ca32bd1] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command HSMGetAllTasksStatusesVDS failed: Not SPM 2017-10-01 16:16:07,831-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler6) [1ca32bd1] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand' return value ' 2017-10-01 16:16:07,831-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler6) [1ca32bd1] HostName = virt4 2017-10-01 16:16:07,831-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler6) [1ca32bd1] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = virt4, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846'})' execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM 2017-10-01 16:16:07,886-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [1ca32bd1] START, ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 4ae85c45 2017-10-01 16:16:07,886-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [1ca32bd1] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:16:08,964-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [1ca32bd1] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:16:08,964-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [1ca32bd1] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:16:08,964-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [1ca32bd1] HostName = virt4 2017-10-01 16:16:08,964-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [1ca32bd1] Command 'ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:16:08,964-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [1ca32bd1] FINISH, ConnectStoragePoolVDSCommand, log id: 4ae85c45 2017-10-01 16:16:08,964-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler6) [1ca32bd1] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:16:09,108-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [1ca32bd1] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:16:09,108-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler6) [1ca32bd1] Command 'SpmStatusVDSCommand(HostName = virt4, SpmStatusVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:16:09,108-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler6) [1ca32bd1] Irs placed on server '5ae29ed2-9c0f-4754-8df3-b43d14726846' failed. Proceed Failover 2017-10-01 16:16:09,153-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [1ca32bd1] START, ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 65a47738 2017-10-01 16:16:09,153-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [1ca32bd1] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:16:09,449-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [1ca32bd1] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:16:09,449-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [1ca32bd1] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:16:09,449-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [1ca32bd1] HostName = virt2 2017-10-01 16:16:09,449-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [1ca32bd1] Command 'ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:16:09,449-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [1ca32bd1] FINISH, ConnectStoragePoolVDSCommand, log id: 65a47738 2017-10-01 16:16:09,449-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler6) [1ca32bd1] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:16:10,543-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [1ca32bd1] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:16:10,543-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler6) [1ca32bd1] Command 'SpmStatusVDSCommand(HostName = virt2, SpmStatusVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:16:10,543-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler6) [1ca32bd1] Irs placed on server 'a0ccfd11-3402-4eba-a119-4987561b738e' failed. Proceed Failover 2017-10-01 16:16:10,587-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [1ca32bd1] START, ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 137ceda0 2017-10-01 16:16:10,588-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [1ca32bd1] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:16:11,697-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [1ca32bd1] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:16:11,697-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [1ca32bd1] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:16:11,697-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [1ca32bd1] HostName = virt3 2017-10-01 16:16:11,697-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [1ca32bd1] Command 'ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:16:11,697-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [1ca32bd1] FINISH, ConnectStoragePoolVDSCommand, log id: 137ceda0 2017-10-01 16:16:11,697-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler6) [1ca32bd1] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:16:11,796-04 INFO [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-34) [19551796] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: c8653dba-63ec-4447-872c-0ffcc6af9c0f Type: Storage 2017-10-01 16:16:11,808-04 WARN [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-34) [19551796] skipping connect and refresh for all hosts, last master 'true', command status 'true' 2017-10-01 16:16:11,825-04 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-6-thread-33) [1ca32bd1] Finished reconstruct for pool 'aca61eac-ed8a-484b-bbee-3645148f6405'. Clearing event queue 2017-10-01 16:16:23,759-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler8) [11f24f37] Command 'GetAllVmStatsVDSCommand(HostName = virt1, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836'})' execution failed: VDSGenericException: VDSNetworkException: Vds timeout occured 2017-10-01 16:16:23,759-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.PollVmStatsRefresher] (DefaultQuartzScheduler8) [11f24f37] Failed to fetch vms info for host 'virt1' - skipping VMs monitoring. 2017-10-01 16:16:23,760-04 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (org.ovirt.thread.pool-6-thread-41) [11f24f37] Server failed to respond, vds_id='45f8b331-842e-48e7-9df8-56adddb93836', vds_name='virt1', vm_count=0, spm_status='None', non-responsive_timeout (seconds)=60, error: VDSGenericException: VDSNetworkException: Vds timeout occured 2017-10-01 16:16:23,781-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-41) [11f24f37] EVENT_ID: VDS_FAILURE(12), Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Vds timeout occured 2017-10-01 16:16:23,783-04 INFO [org.ovirt.engine.core.bll.VdsEventListener] (org.ovirt.thread.pool-6-thread-24) [11f24f37] ResourceManager::vdsNotResponding entered for Host '45f8b331-842e-48e7-9df8-56adddb93836', 'virt1-mgmt' 2017-10-01 16:16:23,790-04 INFO [org.ovirt.engine.core.bll.pm.VdsNotRespondingTreatmentCommand] (org.ovirt.thread.pool-6-thread-24) [1de17e2a] Lock Acquired to object 'EngineLock:{exclusiveLocks='[45f8b331-842e-48e7-9df8-56adddb93836=VDS_FENCE]', sharedLocks=''}' 2017-10-01 16:16:23,795-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [203c6b40] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command HSMGetAllTasksStatusesVDS failed: Not SPM: () 2017-10-01 16:16:23,795-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler6) [203c6b40] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand' return value ' 2017-10-01 16:16:23,795-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler6) [203c6b40] HostName = virt3 2017-10-01 16:16:23,795-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler6) [203c6b40] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = virt3, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736'})' execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () 2017-10-01 16:16:23,826-04 INFO [org.ovirt.engine.core.bll.pm.VdsNotRespondingTreatmentCommand] (org.ovirt.thread.pool-6-thread-24) [1de17e2a] Running command: VdsNotRespondingTreatmentCommand internal: true. Entities affected : ID: 45f8b331-842e-48e7-9df8-56adddb93836 Type: VDS 2017-10-01 16:16:23,843-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [203c6b40] START, ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 2201f761 2017-10-01 16:16:23,843-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [203c6b40] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:16:23,846-04 INFO [org.ovirt.engine.core.bll.pm.SshSoftFencingCommand] (org.ovirt.thread.pool-6-thread-24) [1de17e2a] Running command: SshSoftFencingCommand internal: true. Entities affected : ID: 45f8b331-842e-48e7-9df8-56adddb93836 Type: VDS 2017-10-01 16:16:23,976-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-24) [1de17e2a] EVENT_ID: FENCE_OPERATION_USING_AGENT_AND_PROXY_STARTED(9,020), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Executing power management status on Host virt1 using Proxy Host virt4 and Fence Agent ipmilan:virt1-idrac. 2017-10-01 16:16:23,976-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-6-thread-24) [1de17e2a] START, FenceVdsVDSCommand(HostName = virt4, FenceVdsVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', targetVdsId='45f8b331-842e-48e7-9df8-56adddb93836', action='STATUS', agent='FenceAgent:{id='b92fdde5-0407-4d57-8d00-108a0cea8300', hostId='45f8b331-842e-48e7-9df8-56adddb93836', order='1', type='ipmilan', ip='virt1-idrac', port='null', user='root', password='***', encryptOptions='false', options='privlvl=OPERATOR, delay=10, lanplus=1'}', policy='null'}), log id: 5c035f28 2017-10-01 16:16:23,982-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [203c6b40] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:16:23,982-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [203c6b40] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:16:23,982-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [203c6b40] HostName = virt3 2017-10-01 16:16:23,982-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [203c6b40] Command 'ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:16:23,982-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [203c6b40] FINISH, ConnectStoragePoolVDSCommand, log id: 2201f761 2017-10-01 16:16:23,982-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler6) [203c6b40] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:16:24,087-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler6) [203c6b40] Irs placed on server 'e1e71e9d-a0cd-47db-96c6-f724f6cea736' failed. Proceed Failover 2017-10-01 16:16:24,131-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [203c6b40] START, ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 5af56a25 2017-10-01 16:16:24,131-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [203c6b40] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:16:24,131-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-6-thread-24) [1de17e2a] FINISH, FenceVdsVDSCommand, return: FenceOperationResult:{status='SUCCESS', powerStatus='ON', message=''}, log id: 5c035f28 2017-10-01 16:16:24,131-04 INFO [org.ovirt.engine.core.bll.pm.SshSoftFencingCommand] (org.ovirt.thread.pool-6-thread-24) [1de17e2a] Opening SSH Soft Fencing session on host 'virt1-mgmt' 2017-10-01 16:16:24,489-04 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to virt1-mgmt/192.168.0.34 2017-10-01 16:16:24,494-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [203c6b40] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:16:24,494-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [203c6b40] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:16:24,494-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [203c6b40] HostName = virt4 2017-10-01 16:16:24,494-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [203c6b40] Command 'ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:16:24,494-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [203c6b40] FINISH, ConnectStoragePoolVDSCommand, log id: 5af56a25 2017-10-01 16:16:24,494-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler6) [203c6b40] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:16:24,584-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [203c6b40] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:16:24,584-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler6) [203c6b40] Command 'SpmStatusVDSCommand(HostName = virt4, SpmStatusVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:16:24,584-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler6) [203c6b40] Irs placed on server '5ae29ed2-9c0f-4754-8df3-b43d14726846' failed. Proceed Failover 2017-10-01 16:16:24,627-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [203c6b40] START, ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 6982c585 2017-10-01 16:16:24,627-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [203c6b40] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:16:24,716-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [203c6b40] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:16:24,716-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [203c6b40] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:16:24,716-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [203c6b40] HostName = virt2 2017-10-01 16:16:24,716-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [203c6b40] Command 'ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:16:24,716-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [203c6b40] FINISH, ConnectStoragePoolVDSCommand, log id: 6982c585 2017-10-01 16:16:24,716-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler6) [203c6b40] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:16:24,813-04 INFO [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-8) [143cc1db] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: c8653dba-63ec-4447-872c-0ffcc6af9c0f Type: Storage 2017-10-01 16:16:24,825-04 WARN [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-8) [143cc1db] skipping connect and refresh for all hosts, last master 'true', command status 'true' 2017-10-01 16:16:24,831-04 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-6-thread-40) [203c6b40] Finished reconstruct for pool 'aca61eac-ed8a-484b-bbee-3645148f6405'. Clearing event queue 2017-10-01 16:16:27,145-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler10) [36dba0cb] Command 'GetCapabilitiesVDSCommand(HostName = virt1, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vds='Host[virt1,45f8b331-842e-48e7-9df8-56adddb93836]'})' execution failed: java.net.NoRouteToHostException: No route to host 2017-10-01 16:16:27,145-04 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [36dba0cb] Failure to refresh host 'virt1' runtime info: java.net.NoRouteToHostException: No route to host 2017-10-01 16:16:27,148-04 ERROR [org.ovirt.engine.core.bll.pm.SshSoftFencingCommand] (org.ovirt.thread.pool-6-thread-24) [1de17e2a] SSH Soft Fencing command failed on host 'virt1-mgmt': Failed to get the session. 2017-10-01 16:16:27,152-04 INFO [org.ovirt.engine.core.bll.pm.SshSoftFencingCommand] (org.ovirt.thread.pool-6-thread-24) [1de17e2a] Lock freed to object 'EngineLock:{exclusiveLocks='[45f8b331-842e-48e7-9df8-56adddb93836=VDS_FENCE]', sharedLocks=''}' 2017-10-01 16:16:27,172-04 INFO [org.ovirt.engine.core.bll.pm.VdsKdumpDetectionCommand] (org.ovirt.thread.pool-6-thread-24) [1de17e2a] Running command: VdsKdumpDetectionCommand internal: true. Entities affected : ID: 45f8b331-842e-48e7-9df8-56adddb93836 Type: VDS 2017-10-01 16:16:30,160-04 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to virt1-mgmt/192.168.0.34 2017-10-01 16:16:33,167-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler4) [4093a4c1] Command 'GetCapabilitiesVDSCommand(HostName = virt1, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vds='Host[virt1,45f8b331-842e-48e7-9df8-56adddb93836]'})' execution failed: java.net.NoRouteToHostException: No route to host 2017-10-01 16:16:33,167-04 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [4093a4c1] Failure to refresh host 'virt1' runtime info: java.net.NoRouteToHostException: No route to host 2017-10-01 16:16:35,647-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [974559e] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:16:35,647-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler5) [974559e] Command 'SpmStatusVDSCommand(HostName = virt2, SpmStatusVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:16:36,183-04 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to virt1-mgmt/192.168.0.34 2017-10-01 16:16:36,187-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [974559e] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command HSMGetAllTasksStatusesVDS failed: Not SPM 2017-10-01 16:16:36,187-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler5) [974559e] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand' return value ' 2017-10-01 16:16:36,187-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler5) [974559e] HostName = virt2 2017-10-01 16:16:36,187-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler5) [974559e] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = virt2, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e'})' execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM 2017-10-01 16:16:36,241-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [974559e] START, ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 2ec8aea0 2017-10-01 16:16:36,241-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [974559e] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:16:36,319-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [974559e] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:16:36,319-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [974559e] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:16:36,319-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [974559e] HostName = virt4 2017-10-01 16:16:36,319-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [974559e] Command 'ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:16:36,319-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [974559e] FINISH, ConnectStoragePoolVDSCommand, log id: 2ec8aea0 2017-10-01 16:16:36,319-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler5) [974559e] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:16:36,409-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [974559e] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:16:36,409-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler5) [974559e] Command 'SpmStatusVDSCommand(HostName = virt4, SpmStatusVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:16:36,409-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler5) [974559e] Irs placed on server '5ae29ed2-9c0f-4754-8df3-b43d14726846' failed. Proceed Failover 2017-10-01 16:16:36,453-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [974559e] START, ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 44b07596 2017-10-01 16:16:36,453-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [974559e] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:16:36,540-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [974559e] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:16:36,540-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [974559e] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:16:36,540-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [974559e] HostName = virt2 2017-10-01 16:16:36,540-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [974559e] Command 'ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:16:36,540-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [974559e] FINISH, ConnectStoragePoolVDSCommand, log id: 44b07596 2017-10-01 16:16:36,540-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler5) [974559e] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:16:36,631-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [974559e] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:16:36,632-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler5) [974559e] Command 'SpmStatusVDSCommand(HostName = virt2, SpmStatusVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:16:36,632-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler5) [974559e] Irs placed on server 'a0ccfd11-3402-4eba-a119-4987561b738e' failed. Proceed Failover 2017-10-01 16:16:36,675-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [974559e] START, ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 7146e711 2017-10-01 16:16:36,675-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [974559e] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:16:36,781-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [974559e] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:16:36,781-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [974559e] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:16:36,781-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [974559e] HostName = virt3 2017-10-01 16:16:36,782-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [974559e] Command 'ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:16:36,782-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [974559e] FINISH, ConnectStoragePoolVDSCommand, log id: 7146e711 2017-10-01 16:16:36,782-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler5) [974559e] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:16:36,879-04 INFO [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-11) [25f8f293] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: c8653dba-63ec-4447-872c-0ffcc6af9c0f Type: Storage 2017-10-01 16:16:36,893-04 WARN [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-11) [25f8f293] skipping connect and refresh for all hosts, last master 'true', command status 'true' 2017-10-01 16:16:36,905-04 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-6-thread-10) [974559e] Finished reconstruct for pool 'aca61eac-ed8a-484b-bbee-3645148f6405'. Clearing event queue 2017-10-01 16:16:39,189-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler9) [653b0ca8] Command 'GetCapabilitiesVDSCommand(HostName = virt1, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vds='Host[virt1,45f8b331-842e-48e7-9df8-56adddb93836]'})' execution failed: java.net.NoRouteToHostException: No route to host 2017-10-01 16:16:39,189-04 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [653b0ca8] Failure to refresh host 'virt1' runtime info: java.net.NoRouteToHostException: No route to host 2017-10-01 16:16:42,205-04 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to virt1-mgmt/192.168.0.34 2017-10-01 16:16:45,211-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler9) [7576a9e2] Command 'GetCapabilitiesVDSCommand(HostName = virt1, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vds='Host[virt1,45f8b331-842e-48e7-9df8-56adddb93836]'})' execution failed: java.net.NoRouteToHostException: No route to host 2017-10-01 16:16:45,211-04 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [7576a9e2] Failure to refresh host 'virt1' runtime info: java.net.NoRouteToHostException: No route to host 2017-10-01 16:16:48,226-04 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to virt1-mgmt/192.168.0.34 2017-10-01 16:16:48,230-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [74ab066e] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command HSMGetAllTasksStatusesVDS failed: Not SPM: () 2017-10-01 16:16:48,230-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler7) [74ab066e] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand' return value ' 2017-10-01 16:16:48,230-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler7) [74ab066e] HostName = virt3 2017-10-01 16:16:48,230-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler7) [74ab066e] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = virt3, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736'})' execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () 2017-10-01 16:16:48,284-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [74ab066e] START, ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 77a51f68 2017-10-01 16:16:48,284-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [74ab066e] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:16:48,391-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [74ab066e] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:16:48,391-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [74ab066e] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:16:48,391-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [74ab066e] HostName = virt3 2017-10-01 16:16:48,391-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [74ab066e] Command 'ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:16:48,391-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [74ab066e] FINISH, ConnectStoragePoolVDSCommand, log id: 77a51f68 2017-10-01 16:16:48,391-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler7) [74ab066e] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:16:48,477-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler7) [74ab066e] Irs placed on server 'e1e71e9d-a0cd-47db-96c6-f724f6cea736' failed. Proceed Failover 2017-10-01 16:16:48,521-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [74ab066e] START, ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 6e5db3b7 2017-10-01 16:16:48,521-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [74ab066e] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:16:48,597-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [74ab066e] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:16:48,597-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [74ab066e] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:16:48,597-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [74ab066e] HostName = virt4 2017-10-01 16:16:48,597-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [74ab066e] Command 'ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:16:48,597-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [74ab066e] FINISH, ConnectStoragePoolVDSCommand, log id: 6e5db3b7 2017-10-01 16:16:48,597-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler7) [74ab066e] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:16:48,687-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [74ab066e] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:16:48,687-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler7) [74ab066e] Command 'SpmStatusVDSCommand(HostName = virt4, SpmStatusVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:16:48,687-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler7) [74ab066e] Irs placed on server '5ae29ed2-9c0f-4754-8df3-b43d14726846' failed. Proceed Failover 2017-10-01 16:16:48,731-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [74ab066e] START, ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 96e6df9 2017-10-01 16:16:48,731-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [74ab066e] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:16:48,817-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [74ab066e] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:16:48,817-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [74ab066e] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:16:48,817-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [74ab066e] HostName = virt2 2017-10-01 16:16:48,817-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [74ab066e] Command 'ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:16:48,817-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [74ab066e] FINISH, ConnectStoragePoolVDSCommand, log id: 96e6df9 2017-10-01 16:16:48,817-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler7) [74ab066e] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:16:48,925-04 INFO [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-16) [ccb55f6] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: c8653dba-63ec-4447-872c-0ffcc6af9c0f Type: Storage 2017-10-01 16:16:48,934-04 WARN [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-16) [ccb55f6] skipping connect and refresh for all hosts, last master 'true', command status 'true' 2017-10-01 16:16:48,943-04 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-6-thread-20) [74ab066e] Finished reconstruct for pool 'aca61eac-ed8a-484b-bbee-3645148f6405'. Clearing event queue 2017-10-01 16:16:50,323-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] START, SetVdsStatusVDSCommand(HostName = virt1, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', status='NonResponsive', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 56f07d7e 2017-10-01 16:16:51,231-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler10) [7baf207a] Command 'GetCapabilitiesVDSCommand(HostName = virt1, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vds='Host[virt1,45f8b331-842e-48e7-9df8-56adddb93836]'})' execution failed: java.net.NoRouteToHostException: No route to host 2017-10-01 16:16:51,231-04 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [7baf207a] Failure to refresh host 'virt1' runtime info: java.net.NoRouteToHostException: No route to host 2017-10-01 16:16:51,234-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-6-thread-38) [] FINISH, SetVdsStatusVDSCommand, log id: 56f07d7e 2017-10-01 16:16:54,269-04 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to virt1-mgmt/192.168.0.34 2017-10-01 16:16:56,588-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [420d0ea6] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 16:16:57,201-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-24) [1de17e2a] EVENT_ID: KDUMP_FLOW_NOT_DETECTED_ON_VDS(615), Correlation ID: 1de17e2a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Kdump flow is not in progress on host virt1. 2017-10-01 16:16:57,201-04 WARN [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (org.ovirt.thread.pool-6-thread-24) [1de17e2a] Trying to release exclusive lock which does not exist, lock key: '45f8b331-842e-48e7-9df8-56adddb93836VDS_FENCE' 2017-10-01 16:16:57,203-04 INFO [org.ovirt.engine.core.bll.pm.VdsKdumpDetectionCommand] (org.ovirt.thread.pool-6-thread-24) [1de17e2a] Lock freed to object 'EngineLock:{exclusiveLocks='[45f8b331-842e-48e7-9df8-56adddb93836=VDS_FENCE]', sharedLocks=''}' 2017-10-01 16:16:57,275-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler5) [5922f7b3] Command 'GetCapabilitiesVDSCommand(HostName = virt1, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vds='Host[virt1,45f8b331-842e-48e7-9df8-56adddb93836]'})' execution failed: java.net.NoRouteToHostException: No route to host 2017-10-01 16:16:57,275-04 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [5922f7b3] Failure to refresh host 'virt1' runtime info: java.net.NoRouteToHostException: No route to host 2017-10-01 16:16:57,355-04 INFO [org.ovirt.engine.core.bll.pm.RestartVdsCommand] (org.ovirt.thread.pool-6-thread-24) [] Running command: RestartVdsCommand internal: true. Entities affected : ID: 45f8b331-842e-48e7-9df8-56adddb93836 Type: VDSAction group MANIPULATE_HOST with role type ADMIN 2017-10-01 16:16:57,437-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-24) [] EVENT_ID: FENCE_OPERATION_USING_AGENT_AND_PROXY_STARTED(9,020), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Executing power management status on Host virt1 using Proxy Host virt4 and Fence Agent ipmilan:virt1-idrac. 2017-10-01 16:16:57,438-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] START, FenceVdsVDSCommand(HostName = virt4, FenceVdsVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', targetVdsId='45f8b331-842e-48e7-9df8-56adddb93836', action='STATUS', agent='FenceAgent:{id='b92fdde5-0407-4d57-8d00-108a0cea8300', hostId='45f8b331-842e-48e7-9df8-56adddb93836', order='1', type='ipmilan', ip='virt1-idrac', port='null', user='root', password='***', encryptOptions='false', options='privlvl=OPERATOR, delay=10, lanplus=1'}', policy='null'}), log id: 74d1ad4f 2017-10-01 16:16:58,598-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] FINISH, FenceVdsVDSCommand, return: FenceOperationResult:{status='SUCCESS', powerStatus='ON', message=''}, log id: 74d1ad4f 2017-10-01 16:16:58,603-04 INFO [org.ovirt.engine.core.bll.pm.StopVdsCommand] (org.ovirt.thread.pool-6-thread-24) [6904fe5] Running command: StopVdsCommand internal: true. Entities affected : ID: 45f8b331-842e-48e7-9df8-56adddb93836 Type: VDSAction group MANIPULATE_HOST with role type ADMIN 2017-10-01 16:16:58,605-04 INFO [org.ovirt.engine.core.bll.pm.StopVdsCommand] (org.ovirt.thread.pool-6-thread-24) [6904fe5] Power-Management: STOP of host 'virt1' initiated. 2017-10-01 16:16:58,622-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-24) [6904fe5] EVENT_ID: FENCE_OPERATION_STARTED(9,017), Correlation ID: 6904fe5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Power management stop of Host virt1 initiated. 2017-10-01 16:16:58,637-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-6-thread-24) [6904fe5] START, SetVdsStatusVDSCommand(HostName = virt1, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', status='Reboot', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 6c38381c 2017-10-01 16:16:58,640-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-6-thread-24) [6904fe5] FINISH, SetVdsStatusVDSCommand, log id: 6c38381c 2017-10-01 16:16:58,733-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-24) [6904fe5] EVENT_ID: FENCE_OPERATION_USING_AGENT_AND_PROXY_STARTED(9,020), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Executing power management stop on Host virt1 using Proxy Host virt4 and Fence Agent ipmilan:virt1-idrac. 2017-10-01 16:16:58,733-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-6-thread-24) [6904fe5] START, FenceVdsVDSCommand(HostName = virt4, FenceVdsVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', targetVdsId='45f8b331-842e-48e7-9df8-56adddb93836', action='STOP', agent='FenceAgent:{id='b92fdde5-0407-4d57-8d00-108a0cea8300', hostId='45f8b331-842e-48e7-9df8-56adddb93836', order='1', type='ipmilan', ip='virt1-idrac', port='null', user='root', password='***', encryptOptions='false', options='privlvl=OPERATOR, delay=10, lanplus=1'}', policy='[]'}), log id: 5e7a22d8 2017-10-01 16:16:58,945-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [58eebdb2] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:16:58,945-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler5) [58eebdb2] Command 'SpmStatusVDSCommand(HostName = virt2, SpmStatusVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:16:59,953-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [58eebdb2] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command HSMGetAllTasksStatusesVDS failed: Not SPM 2017-10-01 16:16:59,953-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler5) [58eebdb2] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand' return value ' 2017-10-01 16:16:59,953-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler5) [58eebdb2] HostName = virt2 2017-10-01 16:16:59,953-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler5) [58eebdb2] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = virt2, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e'})' execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM 2017-10-01 16:17:00,006-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [58eebdb2] START, ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 4615a6b7 2017-10-01 16:17:00,006-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [58eebdb2] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:17:00,324-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [58eebdb2] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:17:00,324-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [58eebdb2] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:17:00,324-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [58eebdb2] HostName = virt4 2017-10-01 16:17:00,324-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [58eebdb2] Command 'ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:17:00,325-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [58eebdb2] FINISH, ConnectStoragePoolVDSCommand, log id: 4615a6b7 2017-10-01 16:17:00,325-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler5) [58eebdb2] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:17:01,416-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [58eebdb2] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:17:01,416-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler5) [58eebdb2] Command 'SpmStatusVDSCommand(HostName = virt4, SpmStatusVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:17:01,416-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler5) [58eebdb2] Irs placed on server '5ae29ed2-9c0f-4754-8df3-b43d14726846' failed. Proceed Failover 2017-10-01 16:17:01,459-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [58eebdb2] START, ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 75803d7a 2017-10-01 16:17:01,459-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [58eebdb2] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:17:01,960-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [58eebdb2] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:17:01,960-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [58eebdb2] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:17:01,960-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [58eebdb2] HostName = virt3 2017-10-01 16:17:01,960-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [58eebdb2] Command 'ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:17:01,960-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [58eebdb2] FINISH, ConnectStoragePoolVDSCommand, log id: 75803d7a 2017-10-01 16:17:01,960-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler5) [58eebdb2] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:17:03,048-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler5) [58eebdb2] Irs placed on server 'e1e71e9d-a0cd-47db-96c6-f724f6cea736' failed. Proceed Failover 2017-10-01 16:17:03,091-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [58eebdb2] START, ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 1a8a9493 2017-10-01 16:17:03,092-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [58eebdb2] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:17:07,756-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [58eebdb2] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:17:07,756-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [58eebdb2] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:17:07,756-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [58eebdb2] HostName = virt2 2017-10-01 16:17:07,756-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [58eebdb2] Command 'ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:17:07,756-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [58eebdb2] FINISH, ConnectStoragePoolVDSCommand, log id: 1a8a9493 2017-10-01 16:17:07,756-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler5) [58eebdb2] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:17:07,871-04 INFO [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-21) [3546f37c] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: c8653dba-63ec-4447-872c-0ffcc6af9c0f Type: Storage 2017-10-01 16:17:07,881-04 WARN [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-21) [3546f37c] skipping connect and refresh for all hosts, last master 'true', command status 'true' 2017-10-01 16:17:07,889-04 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-6-thread-37) [58eebdb2] Finished reconstruct for pool 'aca61eac-ed8a-484b-bbee-3645148f6405'. Clearing event queue 2017-10-01 16:17:10,868-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-49) [] starting processDomainRecovery for domain 'f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2:export'. 2017-10-01 16:17:10,923-04 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-49) [] Storage domain 'f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2:export' is not visible to one or more hosts. Since the domain's type is 'ImportExport', hosts status will not be changed to non-operational 2017-10-01 16:17:10,923-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-49) [] Removing vds '[]' from the domain in maintenance cache 2017-10-01 16:17:10,923-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-49) [] Removing host(s) '[]' from hosts unseen domain report cache 2017-10-01 16:17:10,924-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-50) [] starting processDomainRecovery for domain '63d680c7-4c94-4b29-a6cb-f288a128b9bc:iso'. 2017-10-01 16:17:10,979-04 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-50) [] Storage domain '63d680c7-4c94-4b29-a6cb-f288a128b9bc:iso' is not visible to one or more hosts. Since the domain's type is 'ISO', hosts status will not be changed to non-operational 2017-10-01 16:17:10,979-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-50) [] Removing vds '[]' from the domain in maintenance cache 2017-10-01 16:17:10,979-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-50) [] Removing host(s) '[]' from hosts unseen domain report cache 2017-10-01 16:17:12,400-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-6-thread-24) [6904fe5] FINISH, FenceVdsVDSCommand, return: FenceOperationResult:{status='SUCCESS', powerStatus='UNKNOWN', message=''}, log id: 5e7a22d8 2017-10-01 16:17:12,400-04 INFO [org.ovirt.engine.core.bll.pm.SingleAgentFenceActionExecutor] (org.ovirt.thread.pool-6-thread-24) [6904fe5] Waiting for host 'virt1-mgmt' to reach status 'OFF' 2017-10-01 16:17:17,402-04 INFO [org.ovirt.engine.core.bll.pm.SingleAgentFenceActionExecutor] (org.ovirt.thread.pool-6-thread-24) [6904fe5] Attempt 1 to get host 'virt1-mgmt' status 2017-10-01 16:17:17,521-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-24) [6904fe5] EVENT_ID: FENCE_OPERATION_USING_AGENT_AND_PROXY_STARTED(9,020), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Executing power management status on Host virt1 using Proxy Host virt4 and Fence Agent ipmilan:virt1-idrac. 2017-10-01 16:17:17,521-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-6-thread-24) [6904fe5] START, FenceVdsVDSCommand(HostName = virt4, FenceVdsVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', targetVdsId='45f8b331-842e-48e7-9df8-56adddb93836', action='STATUS', agent='FenceAgent:{id='b92fdde5-0407-4d57-8d00-108a0cea8300', hostId='45f8b331-842e-48e7-9df8-56adddb93836', order='1', type='ipmilan', ip='virt1-idrac', port='null', user='root', password='***', encryptOptions='false', options='privlvl=OPERATOR, delay=10, lanplus=1'}', policy='[]'}), log id: 55cb3a13 2017-10-01 16:17:17,847-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-6-thread-24) [6904fe5] FINISH, FenceVdsVDSCommand, return: FenceOperationResult:{status='SUCCESS', powerStatus='OFF', message=''}, log id: 55cb3a13 2017-10-01 16:17:17,847-04 INFO [org.ovirt.engine.core.bll.pm.SingleAgentFenceActionExecutor] (org.ovirt.thread.pool-6-thread-24) [6904fe5] Host 'virt1-mgmt' status is 'OFF' 2017-10-01 16:17:17,886-04 INFO [org.ovirt.engine.core.bll.pm.StopVdsCommand] (org.ovirt.thread.pool-6-thread-24) [6904fe5] Power-Management: STOP host 'virt1' succeeded. 2017-10-01 16:17:17,889-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-24) [6904fe5] EVENT_ID: FENCE_OPERATION_SUCCEEDED(9,018), Correlation ID: 6904fe5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Power management stop of Host virt1 succeeded. 2017-10-01 16:17:17,894-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-24) [6904fe5] EVENT_ID: USER_VDS_STOP(21), Correlation ID: 6904fe5, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host virt1 was stopped by SYSTEM. 2017-10-01 16:17:17,915-04 INFO [org.ovirt.engine.core.bll.storage.FenceVdsManualyCommand] (org.ovirt.thread.pool-6-thread-24) [32363eb9] Running command: FenceVdsManualyCommand internal: true. Entities affected : ID: 45f8b331-842e-48e7-9df8-56adddb93836 Type: VDSAction group MANIPULATE_HOST with role type ADMIN 2017-10-01 16:17:17,938-04 INFO [org.ovirt.engine.core.bll.ClearNonResponsiveVdsVmsCommand] (org.ovirt.thread.pool-6-thread-24) [1cc73bf7] Running command: ClearNonResponsiveVdsVmsCommand internal: true. Entities affected : ID: 45f8b331-842e-48e7-9df8-56adddb93836 Type: VDS 2017-10-01 16:17:17,956-04 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsVMsClearedVDSCommand] (org.ovirt.thread.pool-6-thread-24) [1cc73bf7] START, UpdateVdsVMsClearedVDSCommand(HostName = virt1, UpdateVdsVMsClearedVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836'}), log id: 7f3213d9 2017-10-01 16:17:17,957-04 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsVMsClearedVDSCommand] (org.ovirt.thread.pool-6-thread-24) [1cc73bf7] FINISH, UpdateVdsVMsClearedVDSCommand, log id: 7f3213d9 2017-10-01 16:17:17,963-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-24) [1cc73bf7] EVENT_ID: USER_CLEAR_UNKNOWN_VMS(348), Correlation ID: 1cc73bf7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: All VMs' status on Non Responsive Host virt1 were changed to 'Down' by SYSTEM 2017-10-01 16:17:17,985-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-24) [1cc73bf7] EVENT_ID: VDS_AUTO_FENCE_STATUS(539), Correlation ID: 32363eb9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Auto fence for host virt1 was started. 2017-10-01 16:17:18,062-04 INFO [org.ovirt.engine.core.bll.pm.StartVdsCommand] (org.ovirt.thread.pool-6-thread-24) [] Running command: StartVdsCommand internal: true. Entities affected : ID: 45f8b331-842e-48e7-9df8-56adddb93836 Type: VDSAction group MANIPULATE_HOST with role type ADMIN 2017-10-01 16:17:18,062-04 INFO [org.ovirt.engine.core.bll.pm.StartVdsCommand] (org.ovirt.thread.pool-6-thread-24) [] Power-Management: START of host 'virt1' initiated. 2017-10-01 16:17:18,064-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-24) [] EVENT_ID: FENCE_OPERATION_STARTED(9,017), Correlation ID: 46725ef, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Power management start of Host virt1 initiated. 2017-10-01 16:17:18,143-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-24) [] EVENT_ID: FENCE_OPERATION_USING_AGENT_AND_PROXY_STARTED(9,020), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Executing power management start on Host virt1 using Proxy Host virt4 and Fence Agent ipmilan:virt1-idrac. 2017-10-01 16:17:18,143-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] START, FenceVdsVDSCommand(HostName = virt4, FenceVdsVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', targetVdsId='45f8b331-842e-48e7-9df8-56adddb93836', action='START', agent='FenceAgent:{id='b92fdde5-0407-4d57-8d00-108a0cea8300', hostId='45f8b331-842e-48e7-9df8-56adddb93836', order='1', type='ipmilan', ip='virt1-idrac', port='null', user='root', password='***', encryptOptions='false', options='privlvl=OPERATOR, delay=10, lanplus=1'}', policy='[]'}), log id: 1c82068e 2017-10-01 16:17:18,148-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [ae55aab] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:17:18,148-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler3) [ae55aab] Command 'SpmStatusVDSCommand(HostName = virt2, SpmStatusVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:17:18,255-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [ae55aab] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command HSMGetAllTasksStatusesVDS failed: Not SPM 2017-10-01 16:17:18,255-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler3) [ae55aab] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand' return value ' 2017-10-01 16:17:18,255-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler3) [ae55aab] HostName = virt2 2017-10-01 16:17:18,255-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler3) [ae55aab] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = virt2, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e'})' execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM 2017-10-01 16:17:18,303-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [ae55aab] START, ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 3c721488 2017-10-01 16:17:18,304-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [ae55aab] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:17:19,394-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [ae55aab] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:17:19,394-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [ae55aab] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:17:19,394-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [ae55aab] HostName = virt2 2017-10-01 16:17:19,394-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [ae55aab] Command 'ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:17:19,394-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [ae55aab] FINISH, ConnectStoragePoolVDSCommand, log id: 3c721488 2017-10-01 16:17:19,394-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler3) [ae55aab] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:17:20,486-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [ae55aab] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:17:20,486-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler3) [ae55aab] Command 'SpmStatusVDSCommand(HostName = virt2, SpmStatusVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:17:20,486-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler3) [ae55aab] Irs placed on server 'a0ccfd11-3402-4eba-a119-4987561b738e' failed. Proceed Failover 2017-10-01 16:17:20,530-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [ae55aab] START, ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 5260327c 2017-10-01 16:17:20,530-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [ae55aab] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:17:21,607-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [ae55aab] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:17:21,607-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [ae55aab] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:17:21,607-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [ae55aab] HostName = virt4 2017-10-01 16:17:21,607-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [ae55aab] Command 'ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:17:21,607-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [ae55aab] FINISH, ConnectStoragePoolVDSCommand, log id: 5260327c 2017-10-01 16:17:21,607-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler3) [ae55aab] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:17:21,840-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [ae55aab] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:17:21,840-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler3) [ae55aab] Command 'SpmStatusVDSCommand(HostName = virt4, SpmStatusVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:17:21,840-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler3) [ae55aab] Irs placed on server '5ae29ed2-9c0f-4754-8df3-b43d14726846' failed. Proceed Failover 2017-10-01 16:17:21,884-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [ae55aab] START, ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 658e205b 2017-10-01 16:17:21,884-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [ae55aab] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:17:21,894-04 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-2) [] domain 'f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2:export' in problem 'NOT_REPORTED'. vds: 'virt3' 2017-10-01 16:17:21,895-04 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-2) [] domain '63d680c7-4c94-4b29-a6cb-f288a128b9bc:iso' in problem 'NOT_REPORTED'. vds: 'virt3' 2017-10-01 16:17:22,732-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [ae55aab] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:17:22,732-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [ae55aab] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:17:22,732-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [ae55aab] HostName = virt3 2017-10-01 16:17:22,732-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [ae55aab] Command 'ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:17:22,732-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [ae55aab] FINISH, ConnectStoragePoolVDSCommand, log id: 658e205b 2017-10-01 16:17:22,732-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler3) [ae55aab] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:17:22,832-04 INFO [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-36) [797fece] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: c8653dba-63ec-4447-872c-0ffcc6af9c0f Type: Storage 2017-10-01 16:17:22,844-04 WARN [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-36) [797fece] skipping connect and refresh for all hosts, last master 'true', command status 'true' 2017-10-01 16:17:22,857-04 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-6-thread-32) [ae55aab] Finished reconstruct for pool 'aca61eac-ed8a-484b-bbee-3645148f6405'. Clearing event queue 2017-10-01 16:17:23,729-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] FINISH, FenceVdsVDSCommand, return: FenceOperationResult:{status='SUCCESS', powerStatus='UNKNOWN', message=''}, log id: 1c82068e 2017-10-01 16:17:23,729-04 INFO [org.ovirt.engine.core.bll.pm.SingleAgentFenceActionExecutor] (org.ovirt.thread.pool-6-thread-24) [] Waiting for host 'virt1-mgmt' to reach status 'ON' 2017-10-01 16:17:28,729-04 INFO [org.ovirt.engine.core.bll.pm.SingleAgentFenceActionExecutor] (org.ovirt.thread.pool-6-thread-24) [] Attempt 1 to get host 'virt1-mgmt' status 2017-10-01 16:17:28,808-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-24) [] EVENT_ID: FENCE_OPERATION_USING_AGENT_AND_PROXY_STARTED(9,020), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Executing power management status on Host virt1 using Proxy Host virt4 and Fence Agent ipmilan:virt1-idrac. 2017-10-01 16:17:28,809-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] START, FenceVdsVDSCommand(HostName = virt4, FenceVdsVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', targetVdsId='45f8b331-842e-48e7-9df8-56adddb93836', action='STATUS', agent='FenceAgent:{id='b92fdde5-0407-4d57-8d00-108a0cea8300', hostId='45f8b331-842e-48e7-9df8-56adddb93836', order='1', type='ipmilan', ip='virt1-idrac', port='null', user='root', password='***', encryptOptions='false', options='privlvl=OPERATOR, delay=10, lanplus=1'}', policy='[]'}), log id: 35868028 2017-10-01 16:17:29,913-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (org.ovirt.thread.pool-6-thread-24) [] FINISH, FenceVdsVDSCommand, return: FenceOperationResult:{status='SUCCESS', powerStatus='ON', message=''}, log id: 35868028 2017-10-01 16:17:29,913-04 INFO [org.ovirt.engine.core.bll.pm.SingleAgentFenceActionExecutor] (org.ovirt.thread.pool-6-thread-24) [] Host 'virt1-mgmt' status is 'ON' 2017-10-01 16:17:29,952-04 INFO [org.ovirt.engine.core.bll.pm.StartVdsCommand] (org.ovirt.thread.pool-6-thread-24) [] Power-Management: START host 'virt1' succeeded. 2017-10-01 16:17:29,955-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-24) [] EVENT_ID: FENCE_OPERATION_SUCCEEDED(9,018), Correlation ID: 46725ef, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Power management start of Host virt1 succeeded. 2017-10-01 16:17:29,955-04 INFO [org.ovirt.engine.core.bll.pm.StartVdsCommand] (org.ovirt.thread.pool-6-thread-47) [] Waiting 300 seconds, for server to finish reboot process. 2017-10-01 16:17:29,959-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-24) [] EVENT_ID: USER_VDS_START(20), Correlation ID: 46725ef, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host virt1 was started by SYSTEM. 2017-10-01 16:17:29,964-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-24) [] EVENT_ID: SYSTEM_VDS_RESTART(121), Correlation ID: 1de17e2a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host virt1 was restarted by the engine. 2017-10-01 16:17:29,964-04 WARN [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (org.ovirt.thread.pool-6-thread-24) [] Trying to release exclusive lock which does not exist, lock key: '45f8b331-842e-48e7-9df8-56adddb93836VDS_FENCE' 2017-10-01 16:17:29,964-04 INFO [org.ovirt.engine.core.bll.pm.RestartVdsCommand] (org.ovirt.thread.pool-6-thread-24) [] Lock freed to object 'EngineLock:{exclusiveLocks='[45f8b331-842e-48e7-9df8-56adddb93836=VDS_FENCE]', sharedLocks=''}' 2017-10-01 16:17:29,968-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-24) [] EVENT_ID: VDS_RECOVER(14), Correlation ID: 1de17e2a, Job ID: 626080bd-1b5c-45d2-9670-db051883c701, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host virt1 is rebooting. 2017-10-01 16:17:29,970-04 WARN [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (org.ovirt.thread.pool-6-thread-24) [] Trying to release exclusive lock which does not exist, lock key: '45f8b331-842e-48e7-9df8-56adddb93836VDS_FENCE' 2017-10-01 16:17:29,970-04 INFO [org.ovirt.engine.core.bll.pm.VdsNotRespondingTreatmentCommand] (org.ovirt.thread.pool-6-thread-24) [] Lock freed to object 'EngineLock:{exclusiveLocks='[45f8b331-842e-48e7-9df8-56adddb93836=VDS_FENCE]', sharedLocks=''}' 2017-10-01 16:17:34,864-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [3c64000] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command HSMGetAllTasksStatusesVDS failed: Not SPM: () 2017-10-01 16:17:34,864-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler10) [3c64000] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand' return value ' 2017-10-01 16:17:34,864-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler10) [3c64000] HostName = virt3 2017-10-01 16:17:34,864-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler10) [3c64000] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = virt3, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736'})' execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () 2017-10-01 16:17:34,919-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [3c64000] START, ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 46129f1a 2017-10-01 16:17:34,919-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [3c64000] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:17:36,026-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [3c64000] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:17:36,026-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [3c64000] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:17:36,026-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [3c64000] HostName = virt3 2017-10-01 16:17:36,026-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [3c64000] Command 'ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:17:36,026-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [3c64000] FINISH, ConnectStoragePoolVDSCommand, log id: 46129f1a 2017-10-01 16:17:36,026-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler10) [3c64000] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:17:36,980-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler10) [3c64000] Irs placed on server 'e1e71e9d-a0cd-47db-96c6-f724f6cea736' failed. Proceed Failover 2017-10-01 16:17:37,025-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [3c64000] START, ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 71b39184 2017-10-01 16:17:37,025-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [3c64000] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:17:41,406-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [3c64000] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:17:41,406-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [3c64000] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:17:41,406-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [3c64000] HostName = virt4 2017-10-01 16:17:41,406-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [3c64000] Command 'ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:17:41,406-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [3c64000] FINISH, ConnectStoragePoolVDSCommand, log id: 71b39184 2017-10-01 16:17:41,406-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler10) [3c64000] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:17:42,409-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [3c64000] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:17:42,409-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler10) [3c64000] Command 'SpmStatusVDSCommand(HostName = virt4, SpmStatusVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:17:42,409-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler10) [3c64000] Irs placed on server '5ae29ed2-9c0f-4754-8df3-b43d14726846' failed. Proceed Failover 2017-10-01 16:17:42,453-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [3c64000] START, ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 59dd14f0 2017-10-01 16:17:42,453-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [3c64000] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:17:43,543-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [3c64000] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:17:43,543-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [3c64000] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:17:43,543-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [3c64000] HostName = virt2 2017-10-01 16:17:43,543-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [3c64000] Command 'ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:17:43,543-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [3c64000] FINISH, ConnectStoragePoolVDSCommand, log id: 59dd14f0 2017-10-01 16:17:43,543-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler10) [3c64000] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:17:43,644-04 INFO [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-44) [3238024d] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: c8653dba-63ec-4447-872c-0ffcc6af9c0f Type: Storage 2017-10-01 16:17:43,655-04 WARN [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-44) [3238024d] skipping connect and refresh for all hosts, last master 'true', command status 'true' 2017-10-01 16:17:43,668-04 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-6-thread-29) [3c64000] Finished reconstruct for pool 'aca61eac-ed8a-484b-bbee-3645148f6405'. Clearing event queue 2017-10-01 16:17:54,640-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [7b9db226] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:17:54,640-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler9) [7b9db226] Command 'SpmStatusVDSCommand(HostName = virt2, SpmStatusVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:17:55,649-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [7b9db226] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command HSMGetAllTasksStatusesVDS failed: Not SPM 2017-10-01 16:17:55,649-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler9) [7b9db226] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand' return value ' 2017-10-01 16:17:55,649-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler9) [7b9db226] HostName = virt2 2017-10-01 16:17:55,649-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler9) [7b9db226] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = virt2, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e'})' execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM 2017-10-01 16:17:55,697-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [7b9db226] START, ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: e273b16 2017-10-01 16:17:55,698-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [7b9db226] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:17:56,495-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [7b9db226] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:17:56,495-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [7b9db226] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:17:56,495-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [7b9db226] HostName = virt2 2017-10-01 16:17:56,495-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [7b9db226] Command 'ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:17:56,495-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [7b9db226] FINISH, ConnectStoragePoolVDSCommand, log id: e273b16 2017-10-01 16:17:56,495-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler9) [7b9db226] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:17:56,614-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler7) [2b036d42] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 16:17:57,460-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [7b9db226] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:17:57,460-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler9) [7b9db226] Command 'SpmStatusVDSCommand(HostName = virt2, SpmStatusVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:17:57,460-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler9) [7b9db226] Irs placed on server 'a0ccfd11-3402-4eba-a119-4987561b738e' failed. Proceed Failover 2017-10-01 16:17:57,504-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [7b9db226] START, ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 36e1c0ca 2017-10-01 16:17:57,504-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [7b9db226] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:17:58,581-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [7b9db226] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:17:58,581-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [7b9db226] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:17:58,581-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [7b9db226] HostName = virt4 2017-10-01 16:17:58,581-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [7b9db226] Command 'ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:17:58,581-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [7b9db226] FINISH, ConnectStoragePoolVDSCommand, log id: 36e1c0ca 2017-10-01 16:17:58,581-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler9) [7b9db226] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:17:59,673-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [7b9db226] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:17:59,673-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler9) [7b9db226] Command 'SpmStatusVDSCommand(HostName = virt4, SpmStatusVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:17:59,673-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler9) [7b9db226] Irs placed on server '5ae29ed2-9c0f-4754-8df3-b43d14726846' failed. Proceed Failover 2017-10-01 16:17:59,718-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [7b9db226] START, ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 6414b275 2017-10-01 16:17:59,718-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [7b9db226] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:18:00,825-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [7b9db226] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:18:00,825-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [7b9db226] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:18:00,825-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [7b9db226] HostName = virt3 2017-10-01 16:18:00,825-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [7b9db226] Command 'ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:18:00,825-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [7b9db226] FINISH, ConnectStoragePoolVDSCommand, log id: 6414b275 2017-10-01 16:18:00,825-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler9) [7b9db226] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:18:00,925-04 INFO [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-27) [63c1574d] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: c8653dba-63ec-4447-872c-0ffcc6af9c0f Type: Storage 2017-10-01 16:18:00,932-04 WARN [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-27) [63c1574d] skipping connect and refresh for all hosts, last master 'true', command status 'true' 2017-10-01 16:18:00,941-04 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-6-thread-10) [7b9db226] Finished reconstruct for pool 'aca61eac-ed8a-484b-bbee-3645148f6405'. Clearing event queue 2017-10-01 16:18:12,513-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [6324dd76] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command HSMGetAllTasksStatusesVDS failed: Not SPM: () 2017-10-01 16:18:12,513-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler9) [6324dd76] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand' return value ' 2017-10-01 16:18:12,513-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler9) [6324dd76] HostName = virt3 2017-10-01 16:18:12,513-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler9) [6324dd76] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = virt3, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736'})' execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () 2017-10-01 16:18:12,561-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [6324dd76] START, ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 551ba18d 2017-10-01 16:18:12,562-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [6324dd76] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:18:13,669-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [6324dd76] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:18:13,669-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [6324dd76] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:18:13,669-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [6324dd76] HostName = virt3 2017-10-01 16:18:13,669-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [6324dd76] Command 'ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:18:13,669-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [6324dd76] FINISH, ConnectStoragePoolVDSCommand, log id: 551ba18d 2017-10-01 16:18:13,669-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler9) [6324dd76] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:18:14,755-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler9) [6324dd76] Irs placed on server 'e1e71e9d-a0cd-47db-96c6-f724f6cea736' failed. Proceed Failover 2017-10-01 16:18:14,800-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [6324dd76] START, ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 6bf84a66 2017-10-01 16:18:14,800-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [6324dd76] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:18:15,877-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [6324dd76] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:18:15,877-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [6324dd76] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:18:15,877-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [6324dd76] HostName = virt4 2017-10-01 16:18:15,877-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [6324dd76] Command 'ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:18:15,877-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [6324dd76] FINISH, ConnectStoragePoolVDSCommand, log id: 6bf84a66 2017-10-01 16:18:15,877-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler9) [6324dd76] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:18:16,969-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [6324dd76] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:18:16,969-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler9) [6324dd76] Command 'SpmStatusVDSCommand(HostName = virt4, SpmStatusVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:18:16,969-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler9) [6324dd76] Irs placed on server '5ae29ed2-9c0f-4754-8df3-b43d14726846' failed. Proceed Failover 2017-10-01 16:18:17,013-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [6324dd76] START, ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 5c58f06f 2017-10-01 16:18:17,013-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [6324dd76] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:18:19,098-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [6324dd76] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:18:19,098-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [6324dd76] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:18:19,098-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [6324dd76] HostName = virt2 2017-10-01 16:18:19,098-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [6324dd76] Command 'ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:18:19,098-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [6324dd76] FINISH, ConnectStoragePoolVDSCommand, log id: 5c58f06f 2017-10-01 16:18:19,098-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler9) [6324dd76] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:18:19,198-04 INFO [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-42) [2210a9bd] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: c8653dba-63ec-4447-872c-0ffcc6af9c0f Type: Storage 2017-10-01 16:18:19,207-04 WARN [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-42) [2210a9bd] skipping connect and refresh for all hosts, last master 'true', command status 'true' 2017-10-01 16:18:19,214-04 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-6-thread-20) [6324dd76] Finished reconstruct for pool 'aca61eac-ed8a-484b-bbee-3645148f6405'. Clearing event queue 2017-10-01 16:18:23,740-04 INFO [org.ovirt.engine.core.bll.storage.FenceVdsManualyCommand] (default task-233) [9bfcc396-7a2a-4954-98d5-9669ba1c4120] Lock Acquired to object 'EngineLock:{exclusiveLocks='[45f8b331-842e-48e7-9df8-56adddb93836=VDS_FENCE]', sharedLocks=''}' 2017-10-01 16:18:23,742-04 INFO [org.ovirt.engine.core.bll.storage.FenceVdsManualyCommand] (org.ovirt.thread.pool-6-thread-43) [9bfcc396-7a2a-4954-98d5-9669ba1c4120] Running command: FenceVdsManualyCommand internal: false. Entities affected : ID: 45f8b331-842e-48e7-9df8-56adddb93836 Type: VDSAction group MANIPULATE_HOST with role type ADMIN 2017-10-01 16:18:23,762-04 INFO [org.ovirt.engine.core.bll.ClearNonResponsiveVdsVmsCommand] (org.ovirt.thread.pool-6-thread-43) [5b0f3cca] Running command: ClearNonResponsiveVdsVmsCommand internal: true. Entities affected : ID: 45f8b331-842e-48e7-9df8-56adddb93836 Type: VDS 2017-10-01 16:18:23,778-04 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsVMsClearedVDSCommand] (org.ovirt.thread.pool-6-thread-43) [5b0f3cca] START, UpdateVdsVMsClearedVDSCommand(HostName = virt1, UpdateVdsVMsClearedVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836'}), log id: 40b0f9ef 2017-10-01 16:18:23,778-04 INFO [org.ovirt.engine.core.vdsbroker.UpdateVdsVMsClearedVDSCommand] (org.ovirt.thread.pool-6-thread-43) [5b0f3cca] FINISH, UpdateVdsVMsClearedVDSCommand, log id: 40b0f9ef 2017-10-01 16:18:23,782-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-43) [5b0f3cca] EVENT_ID: USER_CLEAR_UNKNOWN_VMS(348), Correlation ID: 5b0f3cca, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: All VMs' status on Non Responsive Host virt1 were changed to 'Down' by admin@internal 2017-10-01 16:18:23,803-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-43) [5b0f3cca] EVENT_ID: VDS_MANUAL_FENCE_STATUS(494), Correlation ID: 9bfcc396-7a2a-4954-98d5-9669ba1c4120, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Manual fence for host virt1 was started. 2017-10-01 16:18:23,805-04 INFO [org.ovirt.engine.core.bll.storage.FenceVdsManualyCommand] (org.ovirt.thread.pool-6-thread-43) [5b0f3cca] Lock freed to object 'EngineLock:{exclusiveLocks='[45f8b331-842e-48e7-9df8-56adddb93836=VDS_FENCE]', sharedLocks=''}' 2017-10-01 16:18:30,194-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [65bb1555] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:18:30,194-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler2) [65bb1555] Command 'SpmStatusVDSCommand(HostName = virt2, SpmStatusVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:18:31,203-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [65bb1555] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command HSMGetAllTasksStatusesVDS failed: Not SPM 2017-10-01 16:18:31,203-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler2) [65bb1555] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand' return value ' 2017-10-01 16:18:31,203-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler2) [65bb1555] HostName = virt2 2017-10-01 16:18:31,203-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler2) [65bb1555] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = virt2, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e'})' execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM 2017-10-01 16:18:31,251-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [65bb1555] START, ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 7d1f1ca7 2017-10-01 16:18:31,251-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [65bb1555] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:18:32,327-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [65bb1555] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:18:32,327-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [65bb1555] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:18:32,327-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [65bb1555] HostName = virt4 2017-10-01 16:18:32,327-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [65bb1555] Command 'ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:18:32,327-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [65bb1555] FINISH, ConnectStoragePoolVDSCommand, log id: 7d1f1ca7 2017-10-01 16:18:32,327-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler2) [65bb1555] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:18:33,426-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [65bb1555] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:18:33,426-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler2) [65bb1555] Command 'SpmStatusVDSCommand(HostName = virt4, SpmStatusVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:18:33,426-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler2) [65bb1555] Irs placed on server '5ae29ed2-9c0f-4754-8df3-b43d14726846' failed. Proceed Failover 2017-10-01 16:18:33,476-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [65bb1555] START, ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 3bf8e632 2017-10-01 16:18:33,476-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [65bb1555] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:18:33,779-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [65bb1555] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:18:33,779-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [65bb1555] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:18:33,779-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [65bb1555] HostName = virt3 2017-10-01 16:18:33,779-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [65bb1555] Command 'ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:18:33,779-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [65bb1555] FINISH, ConnectStoragePoolVDSCommand, log id: 3bf8e632 2017-10-01 16:18:33,779-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler2) [65bb1555] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:18:34,865-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler2) [65bb1555] Irs placed on server 'e1e71e9d-a0cd-47db-96c6-f724f6cea736' failed. Proceed Failover 2017-10-01 16:18:34,910-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [65bb1555] START, ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 235be306 2017-10-01 16:18:34,910-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [65bb1555] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:18:35,202-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [65bb1555] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:18:35,202-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [65bb1555] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:18:35,202-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [65bb1555] HostName = virt2 2017-10-01 16:18:35,202-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [65bb1555] Command 'ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:18:35,202-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [65bb1555] FINISH, ConnectStoragePoolVDSCommand, log id: 235be306 2017-10-01 16:18:35,202-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler2) [65bb1555] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:18:35,308-04 INFO [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-4) [44590649] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: c8653dba-63ec-4447-872c-0ffcc6af9c0f Type: Storage 2017-10-01 16:18:35,323-04 WARN [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-4) [44590649] skipping connect and refresh for all hosts, last master 'true', command status 'true' 2017-10-01 16:18:35,331-04 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-6-thread-39) [65bb1555] Finished reconstruct for pool 'aca61eac-ed8a-484b-bbee-3645148f6405'. Clearing event queue 2017-10-01 16:18:46,298-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3c3bf510] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:18:46,298-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler2) [3c3bf510] Command 'SpmStatusVDSCommand(HostName = virt2, SpmStatusVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:18:47,308-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3c3bf510] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command HSMGetAllTasksStatusesVDS failed: Not SPM 2017-10-01 16:18:47,308-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler2) [3c3bf510] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand' return value ' 2017-10-01 16:18:47,308-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler2) [3c3bf510] HostName = virt2 2017-10-01 16:18:47,308-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler2) [3c3bf510] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = virt2, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e'})' execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM 2017-10-01 16:18:47,357-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [3c3bf510] START, ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 2844f9e3 2017-10-01 16:18:47,357-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [3c3bf510] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:18:50,417-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3c3bf510] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:18:50,417-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [3c3bf510] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:18:50,417-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [3c3bf510] HostName = virt3 2017-10-01 16:18:50,417-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [3c3bf510] Command 'ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:18:50,417-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [3c3bf510] FINISH, ConnectStoragePoolVDSCommand, log id: 2844f9e3 2017-10-01 16:18:50,417-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler2) [3c3bf510] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:18:51,292-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler2) [3c3bf510] Irs placed on server 'e1e71e9d-a0cd-47db-96c6-f724f6cea736' failed. Proceed Failover 2017-10-01 16:18:51,337-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [3c3bf510] START, ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 3a935389 2017-10-01 16:18:51,337-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [3c3bf510] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:18:52,324-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3c3bf510] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:18:52,324-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [3c3bf510] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:18:52,324-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [3c3bf510] HostName = virt4 2017-10-01 16:18:52,324-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [3c3bf510] Command 'ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:18:52,324-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [3c3bf510] FINISH, ConnectStoragePoolVDSCommand, log id: 3a935389 2017-10-01 16:18:52,324-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler2) [3c3bf510] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:18:53,418-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3c3bf510] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:18:53,418-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler2) [3c3bf510] Command 'SpmStatusVDSCommand(HostName = virt4, SpmStatusVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:18:53,418-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler2) [3c3bf510] Irs placed on server '5ae29ed2-9c0f-4754-8df3-b43d14726846' failed. Proceed Failover 2017-10-01 16:18:53,462-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [3c3bf510] START, ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 5f9731ab 2017-10-01 16:18:53,462-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [3c3bf510] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:18:54,552-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [3c3bf510] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:18:54,552-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [3c3bf510] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:18:54,552-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [3c3bf510] HostName = virt2 2017-10-01 16:18:54,552-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [3c3bf510] Command 'ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:18:54,552-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler2) [3c3bf510] FINISH, ConnectStoragePoolVDSCommand, log id: 5f9731ab 2017-10-01 16:18:54,552-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler2) [3c3bf510] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:18:54,650-04 INFO [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-33) [5ef40248] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: c8653dba-63ec-4447-872c-0ffcc6af9c0f Type: Storage 2017-10-01 16:18:54,658-04 WARN [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-33) [5ef40248] skipping connect and refresh for all hosts, last master 'true', command status 'true' 2017-10-01 16:18:54,666-04 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-6-thread-2) [3c3bf510] Finished reconstruct for pool 'aca61eac-ed8a-484b-bbee-3645148f6405'. Clearing event queue 2017-10-01 16:18:56,636-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [6fbbc9de] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 16:19:05,647-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [7d863410] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:19:05,648-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler5) [7d863410] Command 'SpmStatusVDSCommand(HostName = virt2, SpmStatusVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:19:06,389-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [7d863410] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command HSMGetAllTasksStatusesVDS failed: Not SPM 2017-10-01 16:19:06,389-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler5) [7d863410] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand' return value ' 2017-10-01 16:19:06,389-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler5) [7d863410] HostName = virt2 2017-10-01 16:19:06,389-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler5) [7d863410] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = virt2, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e'})' execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM 2017-10-01 16:19:06,437-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [7d863410] START, ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 5e4dd981 2017-10-01 16:19:06,437-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [7d863410] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:19:07,500-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [7d863410] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:19:07,500-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [7d863410] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:19:07,500-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [7d863410] HostName = virt2 2017-10-01 16:19:07,500-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [7d863410] Command 'ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:19:07,500-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [7d863410] FINISH, ConnectStoragePoolVDSCommand, log id: 5e4dd981 2017-10-01 16:19:07,500-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler5) [7d863410] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:19:08,594-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [7d863410] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:19:08,594-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler5) [7d863410] Command 'SpmStatusVDSCommand(HostName = virt2, SpmStatusVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:19:08,594-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler5) [7d863410] Irs placed on server 'a0ccfd11-3402-4eba-a119-4987561b738e' failed. Proceed Failover 2017-10-01 16:19:08,638-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [7d863410] START, ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 2c7a95d8 2017-10-01 16:19:08,639-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [7d863410] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:19:09,716-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [7d863410] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:19:09,716-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [7d863410] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:19:09,716-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [7d863410] HostName = virt4 2017-10-01 16:19:09,716-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [7d863410] Command 'ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:19:09,716-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [7d863410] FINISH, ConnectStoragePoolVDSCommand, log id: 2c7a95d8 2017-10-01 16:19:09,716-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler5) [7d863410] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:19:10,808-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [7d863410] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:19:10,808-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler5) [7d863410] Command 'SpmStatusVDSCommand(HostName = virt4, SpmStatusVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:19:10,808-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler5) [7d863410] Irs placed on server '5ae29ed2-9c0f-4754-8df3-b43d14726846' failed. Proceed Failover 2017-10-01 16:19:10,852-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [7d863410] START, ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 4ac2d41b 2017-10-01 16:19:10,852-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [7d863410] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:19:11,961-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [7d863410] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:19:11,961-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [7d863410] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:19:11,961-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [7d863410] HostName = virt3 2017-10-01 16:19:11,961-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [7d863410] Command 'ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:19:11,961-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [7d863410] FINISH, ConnectStoragePoolVDSCommand, log id: 4ac2d41b 2017-10-01 16:19:11,961-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler5) [7d863410] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:19:12,060-04 INFO [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-28) [5ed5392c] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: c8653dba-63ec-4447-872c-0ffcc6af9c0f Type: Storage 2017-10-01 16:19:12,067-04 WARN [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-28) [5ed5392c] skipping connect and refresh for all hosts, last master 'true', command status 'true' 2017-10-01 16:19:12,073-04 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-6-thread-41) [7d863410] Finished reconstruct for pool 'aca61eac-ed8a-484b-bbee-3645148f6405'. Clearing event queue 2017-10-01 16:19:22,681-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [7687e879] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command HSMGetAllTasksStatusesVDS failed: Not SPM: () 2017-10-01 16:19:22,681-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler5) [7687e879] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand' return value ' 2017-10-01 16:19:22,681-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler5) [7687e879] HostName = virt3 2017-10-01 16:19:22,681-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler5) [7687e879] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = virt3, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736'})' execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () 2017-10-01 16:19:22,729-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [7687e879] START, ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 37dbfdf4 2017-10-01 16:19:22,729-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [7687e879] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:19:23,837-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [7687e879] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:19:23,837-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [7687e879] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:19:23,837-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [7687e879] HostName = virt3 2017-10-01 16:19:23,837-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [7687e879] Command 'ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:19:23,837-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [7687e879] FINISH, ConnectStoragePoolVDSCommand, log id: 37dbfdf4 2017-10-01 16:19:23,837-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler5) [7687e879] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:19:24,929-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler5) [7687e879] Irs placed on server 'e1e71e9d-a0cd-47db-96c6-f724f6cea736' failed. Proceed Failover 2017-10-01 16:19:24,974-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [7687e879] START, ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 5cb339cf 2017-10-01 16:19:24,974-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [7687e879] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:19:26,051-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [7687e879] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:19:26,051-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [7687e879] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:19:26,051-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [7687e879] HostName = virt4 2017-10-01 16:19:26,051-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [7687e879] Command 'ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:19:26,051-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [7687e879] FINISH, ConnectStoragePoolVDSCommand, log id: 5cb339cf 2017-10-01 16:19:26,051-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler5) [7687e879] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:19:26,056-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] START, SetVdsStatusVDSCommand(HostName = virt1, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', status='NonResponsive', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 159a4456 2017-10-01 16:19:26,065-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-6-thread-3) [] FINISH, SetVdsStatusVDSCommand, log id: 159a4456 2017-10-01 16:19:27,143-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [7687e879] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:19:27,143-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler5) [7687e879] Command 'SpmStatusVDSCommand(HostName = virt4, SpmStatusVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:19:27,143-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler5) [7687e879] Irs placed on server '5ae29ed2-9c0f-4754-8df3-b43d14726846' failed. Proceed Failover 2017-10-01 16:19:27,194-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [7687e879] START, ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 23e67959 2017-10-01 16:19:27,194-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [7687e879] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:19:27,275-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler4) [741d8ccd] Command 'GetAllVmStatsVDSCommand(HostName = virt1, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836'})' execution failed: VDSGenericException: VDSNetworkException: Vds timeout occured 2017-10-01 16:19:27,277-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.PollVmStatsRefresher] (DefaultQuartzScheduler4) [741d8ccd] Failed to fetch vms info for host 'virt1' - skipping VMs monitoring. 2017-10-01 16:19:27,778-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [7687e879] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:19:27,779-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [7687e879] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:19:27,779-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [7687e879] HostName = virt2 2017-10-01 16:19:27,779-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [7687e879] Command 'ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:19:27,779-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [7687e879] FINISH, ConnectStoragePoolVDSCommand, log id: 23e67959 2017-10-01 16:19:27,779-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler5) [7687e879] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:19:27,884-04 INFO [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-46) [6c915e7b] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: c8653dba-63ec-4447-872c-0ffcc6af9c0f Type: Storage 2017-10-01 16:19:27,891-04 WARN [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-46) [6c915e7b] skipping connect and refresh for all hosts, last master 'true', command status 'true' 2017-10-01 16:19:27,906-04 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-6-thread-11) [7687e879] Finished reconstruct for pool 'aca61eac-ed8a-484b-bbee-3645148f6405'. Clearing event queue 2017-10-01 16:19:28,032-04 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to virt1-mgmt/192.168.0.34 2017-10-01 16:19:31,039-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler10) [5ea66ce2] Command 'GetCapabilitiesVDSCommand(HostName = virt1, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vds='Host[virt1,45f8b331-842e-48e7-9df8-56adddb93836]'})' execution failed: java.net.NoRouteToHostException: No route to host 2017-10-01 16:19:31,039-04 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler10) [5ea66ce2] Failure to refresh host 'virt1' runtime info: java.net.NoRouteToHostException: No route to host 2017-10-01 16:19:34,054-04 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to virt1-mgmt/192.168.0.34 2017-10-01 16:19:37,061-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler1) [415fad2f] Command 'GetCapabilitiesVDSCommand(HostName = virt1, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vds='Host[virt1,45f8b331-842e-48e7-9df8-56adddb93836]'})' execution failed: java.net.NoRouteToHostException: No route to host 2017-10-01 16:19:37,061-04 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [415fad2f] Failure to refresh host 'virt1' runtime info: java.net.NoRouteToHostException: No route to host 2017-10-01 16:19:38,880-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [172b6ac0] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:19:38,880-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler9) [172b6ac0] Command 'SpmStatusVDSCommand(HostName = virt2, SpmStatusVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:19:39,888-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [172b6ac0] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command HSMGetAllTasksStatusesVDS failed: Not SPM 2017-10-01 16:19:39,888-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler9) [172b6ac0] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand' return value ' 2017-10-01 16:19:39,888-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler9) [172b6ac0] HostName = virt2 2017-10-01 16:19:39,888-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler9) [172b6ac0] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = virt2, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e'})' execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM 2017-10-01 16:19:39,937-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [172b6ac0] START, ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 4678516f 2017-10-01 16:19:39,937-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [172b6ac0] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:19:40,076-04 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to virt1-mgmt/192.168.0.34 2017-10-01 16:19:43,083-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler3) [30af3945] Command 'GetCapabilitiesVDSCommand(HostName = virt1, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vds='Host[virt1,45f8b331-842e-48e7-9df8-56adddb93836]'})' execution failed: java.net.NoRouteToHostException: No route to host 2017-10-01 16:19:43,083-04 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [30af3945] Failure to refresh host 'virt1' runtime info: java.net.NoRouteToHostException: No route to host 2017-10-01 16:19:44,804-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [172b6ac0] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:19:44,804-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [172b6ac0] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:19:44,804-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [172b6ac0] HostName = virt2 2017-10-01 16:19:44,804-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [172b6ac0] Command 'ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:19:44,804-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [172b6ac0] FINISH, ConnectStoragePoolVDSCommand, log id: 4678516f 2017-10-01 16:19:44,804-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler9) [172b6ac0] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:19:45,586-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [172b6ac0] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:19:45,586-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler9) [172b6ac0] Command 'SpmStatusVDSCommand(HostName = virt2, SpmStatusVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:19:45,586-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler9) [172b6ac0] Irs placed on server 'a0ccfd11-3402-4eba-a119-4987561b738e' failed. Proceed Failover 2017-10-01 16:19:45,630-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [172b6ac0] START, ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 51589546 2017-10-01 16:19:45,630-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [172b6ac0] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:19:46,098-04 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to virt1-mgmt/192.168.0.34 2017-10-01 16:19:46,102-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [172b6ac0] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:19:46,102-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [172b6ac0] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:19:46,102-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [172b6ac0] HostName = virt3 2017-10-01 16:19:46,102-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [172b6ac0] Command 'ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:19:46,102-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [172b6ac0] FINISH, ConnectStoragePoolVDSCommand, log id: 51589546 2017-10-01 16:19:46,102-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler9) [172b6ac0] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:19:46,198-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler9) [172b6ac0] Irs placed on server 'e1e71e9d-a0cd-47db-96c6-f724f6cea736' failed. Proceed Failover 2017-10-01 16:19:46,242-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [172b6ac0] START, ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 2927d54 2017-10-01 16:19:46,242-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [172b6ac0] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:19:46,318-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [172b6ac0] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:19:46,318-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [172b6ac0] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:19:46,318-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [172b6ac0] HostName = virt4 2017-10-01 16:19:46,318-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [172b6ac0] Command 'ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:19:46,318-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [172b6ac0] FINISH, ConnectStoragePoolVDSCommand, log id: 2927d54 2017-10-01 16:19:46,318-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler9) [172b6ac0] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:19:46,417-04 INFO [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-17) [6aed] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: c8653dba-63ec-4447-872c-0ffcc6af9c0f Type: Storage 2017-10-01 16:19:46,425-04 WARN [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-17) [6aed] skipping connect and refresh for all hosts, last master 'true', command status 'true' 2017-10-01 16:19:46,432-04 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-6-thread-31) [172b6ac0] Finished reconstruct for pool 'aca61eac-ed8a-484b-bbee-3645148f6405'. Clearing event queue 2017-10-01 16:19:49,103-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler1) [71bfca65] Command 'GetCapabilitiesVDSCommand(HostName = virt1, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vds='Host[virt1,45f8b331-842e-48e7-9df8-56adddb93836]'})' execution failed: java.net.NoRouteToHostException: No route to host 2017-10-01 16:19:49,103-04 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [71bfca65] Failure to refresh host 'virt1' runtime info: java.net.NoRouteToHostException: No route to host 2017-10-01 16:19:52,118-04 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to virt1-mgmt/192.168.0.34 2017-10-01 16:19:55,123-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler3) [67adc5cd] Command 'GetCapabilitiesVDSCommand(HostName = virt1, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vds='Host[virt1,45f8b331-842e-48e7-9df8-56adddb93836]'})' execution failed: java.net.NoRouteToHostException: No route to host 2017-10-01 16:19:55,123-04 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [67adc5cd] Failure to refresh host 'virt1' runtime info: java.net.NoRouteToHostException: No route to host 2017-10-01 16:19:56,656-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [1a761be] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 16:19:57,449-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [9e1f832] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:19:57,449-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler1) [9e1f832] Command 'SpmStatusVDSCommand(HostName = virt4, SpmStatusVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:19:58,139-04 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to virt1-mgmt/192.168.0.34 2017-10-01 16:19:58,143-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [9e1f832] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command HSMGetAllTasksStatusesVDS failed: Not SPM 2017-10-01 16:19:58,143-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler1) [9e1f832] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand' return value ' 2017-10-01 16:19:58,143-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler1) [9e1f832] HostName = virt4 2017-10-01 16:19:58,143-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler1) [9e1f832] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = virt4, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846'})' execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM 2017-10-01 16:19:58,191-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [9e1f832] START, ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 247bf2c6 2017-10-01 16:19:58,192-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [9e1f832] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:19:58,279-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [9e1f832] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:19:58,279-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [9e1f832] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:19:58,279-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [9e1f832] HostName = virt2 2017-10-01 16:19:58,279-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [9e1f832] Command 'ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:19:58,279-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [9e1f832] FINISH, ConnectStoragePoolVDSCommand, log id: 247bf2c6 2017-10-01 16:19:58,279-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler1) [9e1f832] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:19:58,371-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [9e1f832] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:19:58,371-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler1) [9e1f832] Command 'SpmStatusVDSCommand(HostName = virt2, SpmStatusVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:19:58,371-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler1) [9e1f832] Irs placed on server 'a0ccfd11-3402-4eba-a119-4987561b738e' failed. Proceed Failover 2017-10-01 16:19:58,415-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [9e1f832] START, ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 7e393d62 2017-10-01 16:19:58,415-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [9e1f832] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:19:58,491-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [9e1f832] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:19:58,491-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [9e1f832] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:19:58,491-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [9e1f832] HostName = virt4 2017-10-01 16:19:58,491-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [9e1f832] Command 'ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:19:58,491-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [9e1f832] FINISH, ConnectStoragePoolVDSCommand, log id: 7e393d62 2017-10-01 16:19:58,491-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler1) [9e1f832] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:19:58,583-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [9e1f832] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:19:58,583-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler1) [9e1f832] Command 'SpmStatusVDSCommand(HostName = virt4, SpmStatusVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:19:58,583-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler1) [9e1f832] Irs placed on server '5ae29ed2-9c0f-4754-8df3-b43d14726846' failed. Proceed Failover 2017-10-01 16:19:58,627-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [9e1f832] START, ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: a246233 2017-10-01 16:19:58,627-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [9e1f832] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:19:58,732-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [9e1f832] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:19:58,732-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [9e1f832] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:19:58,732-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [9e1f832] HostName = virt3 2017-10-01 16:19:58,732-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [9e1f832] Command 'ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:19:58,732-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [9e1f832] FINISH, ConnectStoragePoolVDSCommand, log id: a246233 2017-10-01 16:19:58,732-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler1) [9e1f832] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:19:58,832-04 INFO [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-30) [36a9bb35] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: c8653dba-63ec-4447-872c-0ffcc6af9c0f Type: Storage 2017-10-01 16:19:58,839-04 WARN [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-30) [36a9bb35] skipping connect and refresh for all hosts, last master 'true', command status 'true' 2017-10-01 16:19:58,850-04 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-6-thread-21) [9e1f832] Finished reconstruct for pool 'aca61eac-ed8a-484b-bbee-3645148f6405'. Clearing event queue 2017-10-01 16:20:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [151f3053] Autorecovering 1 storage domains 2017-10-01 16:20:00,009-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler7) [151f3053] Autorecovering storage domains id: c8653dba-63ec-4447-872c-0ffcc6af9c0f 2017-10-01 16:20:00,013-04 INFO [org.ovirt.engine.core.bll.storage.connection.ConnectDomainToStorageCommand] (DefaultQuartzScheduler7) [2334fefe] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: c8653dba-63ec-4447-872c-0ffcc6af9c0f Type: Storage 2017-10-01 16:20:00,013-04 INFO [org.ovirt.engine.core.bll.storage.connection.ConnectDomainToStorageCommand] (DefaultQuartzScheduler7) [2334fefe] ConnectDomainToStorage. Before Connect all hosts to pool. Time: Sun Oct 01 16:20:00 EDT 2017 2017-10-01 16:20:00,067-04 INFO [org.ovirt.engine.core.bll.storage.connection.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-6-thread-6) [6693f890] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN 2017-10-01 16:20:00,067-04 INFO [org.ovirt.engine.core.bll.storage.connection.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-6-thread-35) [3c289f97] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN 2017-10-01 16:20:00,070-04 INFO [org.ovirt.engine.core.bll.storage.connection.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-6-thread-50) [3c67d681] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN 2017-10-01 16:20:00,085-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-6) [6693f890] START, ConnectStorageServerVDSCommand(HostName = virt2, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='00000000-0000-0000-0000-000000000000', storageType='NFS', connectionList='[StorageServerConnections:{id='5e557659-6817-40fa-9a55-7458125404a6', connection='virtfs-fs:/nfs/data', iqn='null', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 78d611c7 2017-10-01 16:20:00,085-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-50) [3c67d681] START, ConnectStorageServerVDSCommand(HostName = virt4, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='00000000-0000-0000-0000-000000000000', storageType='NFS', connectionList='[StorageServerConnections:{id='5e557659-6817-40fa-9a55-7458125404a6', connection='virtfs-fs:/nfs/data', iqn='null', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 4268e8d0 2017-10-01 16:20:00,097-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-50) [3c67d681] FINISH, ConnectStorageServerVDSCommand, return: {5e557659-6817-40fa-9a55-7458125404a6=0}, log id: 4268e8d0 2017-10-01 16:20:00,099-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-6) [6693f890] FINISH, ConnectStorageServerVDSCommand, return: {5e557659-6817-40fa-9a55-7458125404a6=0}, log id: 78d611c7 2017-10-01 16:20:00,106-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-35) [3c289f97] START, ConnectStorageServerVDSCommand(HostName = virt3, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='00000000-0000-0000-0000-000000000000', storageType='NFS', connectionList='[StorageServerConnections:{id='5e557659-6817-40fa-9a55-7458125404a6', connection='virtfs-fs:/nfs/data', iqn='null', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 7d6fcb64 2017-10-01 16:20:00,117-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-35) [3c289f97] FINISH, ConnectStorageServerVDSCommand, return: {5e557659-6817-40fa-9a55-7458125404a6=0}, log id: 7d6fcb64 2017-10-01 16:20:00,119-04 INFO [org.ovirt.engine.core.bll.storage.connection.ConnectDomainToStorageCommand] (DefaultQuartzScheduler7) [2334fefe] ConnectDomainToStorage. After Connect all hosts to pool. Time: Sun Oct 01 16:20:00 EDT 2017 2017-10-01 16:20:01,145-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler6) [37ecd067] Command 'GetCapabilitiesVDSCommand(HostName = virt1, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vds='Host[virt1,45f8b331-842e-48e7-9df8-56adddb93836]'})' execution failed: java.net.NoRouteToHostException: No route to host 2017-10-01 16:20:01,145-04 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [37ecd067] Failure to refresh host 'virt1' runtime info: java.net.NoRouteToHostException: No route to host 2017-10-01 16:20:04,161-04 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to virt1-mgmt/192.168.0.34 2017-10-01 16:20:07,167-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler9) [5f5e6b43] Command 'GetCapabilitiesVDSCommand(HostName = virt1, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vds='Host[virt1,45f8b331-842e-48e7-9df8-56adddb93836]'})' execution failed: java.net.NoRouteToHostException: No route to host 2017-10-01 16:20:07,167-04 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [5f5e6b43] Failure to refresh host 'virt1' runtime info: java.net.NoRouteToHostException: No route to host 2017-10-01 16:20:10,182-04 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to virt1-mgmt/192.168.0.34 2017-10-01 16:20:10,187-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [18d96beb] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command HSMGetAllTasksStatusesVDS failed: Not SPM: () 2017-10-01 16:20:10,187-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler9) [18d96beb] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand' return value ' 2017-10-01 16:20:10,189-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler9) [18d96beb] HostName = virt3 2017-10-01 16:20:10,189-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler9) [18d96beb] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = virt3, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736'})' execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () 2017-10-01 16:20:10,244-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [18d96beb] START, ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 527bf42e 2017-10-01 16:20:10,245-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [18d96beb] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:20:10,334-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [18d96beb] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:20:10,334-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [18d96beb] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:20:10,334-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [18d96beb] HostName = virt2 2017-10-01 16:20:10,334-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [18d96beb] Command 'ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:20:10,334-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [18d96beb] FINISH, ConnectStoragePoolVDSCommand, log id: 527bf42e 2017-10-01 16:20:10,334-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler9) [18d96beb] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:20:10,427-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [18d96beb] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:20:10,427-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler9) [18d96beb] Command 'SpmStatusVDSCommand(HostName = virt2, SpmStatusVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:20:10,427-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler9) [18d96beb] Irs placed on server 'a0ccfd11-3402-4eba-a119-4987561b738e' failed. Proceed Failover 2017-10-01 16:20:10,471-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [18d96beb] START, ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 265a7be5 2017-10-01 16:20:10,471-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [18d96beb] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:20:10,588-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [18d96beb] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:20:10,588-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [18d96beb] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:20:10,588-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [18d96beb] HostName = virt3 2017-10-01 16:20:10,588-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [18d96beb] Command 'ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:20:10,588-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [18d96beb] FINISH, ConnectStoragePoolVDSCommand, log id: 265a7be5 2017-10-01 16:20:10,588-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler9) [18d96beb] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:20:10,673-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler9) [18d96beb] Irs placed on server 'e1e71e9d-a0cd-47db-96c6-f724f6cea736' failed. Proceed Failover 2017-10-01 16:20:10,716-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [18d96beb] START, ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 590b8407 2017-10-01 16:20:10,716-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [18d96beb] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:20:10,790-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [18d96beb] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:20:10,791-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [18d96beb] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:20:10,791-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [18d96beb] HostName = virt4 2017-10-01 16:20:10,791-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [18d96beb] Command 'ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:20:10,791-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler9) [18d96beb] FINISH, ConnectStoragePoolVDSCommand, log id: 590b8407 2017-10-01 16:20:10,791-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler9) [18d96beb] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:20:10,894-04 INFO [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-48) [3c5aef8] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: c8653dba-63ec-4447-872c-0ffcc6af9c0f Type: Storage 2017-10-01 16:20:10,909-04 WARN [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-48) [3c5aef8] skipping connect and refresh for all hosts, last master 'true', command status 'true' 2017-10-01 16:20:10,915-04 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-6-thread-36) [18d96beb] Finished reconstruct for pool 'aca61eac-ed8a-484b-bbee-3645148f6405'. Clearing event queue 2017-10-01 16:20:13,189-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler1) [11c21a97] Command 'GetCapabilitiesVDSCommand(HostName = virt1, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vds='Host[virt1,45f8b331-842e-48e7-9df8-56adddb93836]'})' execution failed: java.net.NoRouteToHostException: No route to host 2017-10-01 16:20:13,189-04 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler1) [11c21a97] Failure to refresh host 'virt1' runtime info: java.net.NoRouteToHostException: No route to host 2017-10-01 16:20:16,204-04 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to virt1-mgmt/192.168.0.34 2017-10-01 16:20:19,211-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler7) [55983517] Command 'GetCapabilitiesVDSCommand(HostName = virt1, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vds='Host[virt1,45f8b331-842e-48e7-9df8-56adddb93836]'})' execution failed: java.net.NoRouteToHostException: No route to host 2017-10-01 16:20:19,211-04 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [55983517] Failure to refresh host 'virt1' runtime info: java.net.NoRouteToHostException: No route to host 2017-10-01 16:20:21,663-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [149c59d8] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:20:21,664-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler4) [149c59d8] Command 'SpmStatusVDSCommand(HostName = virt4, SpmStatusVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:20:22,226-04 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to virt1-mgmt/192.168.0.34 2017-10-01 16:20:22,230-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [149c59d8] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command HSMGetAllTasksStatusesVDS failed: Not SPM 2017-10-01 16:20:22,230-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler4) [149c59d8] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand' return value ' 2017-10-01 16:20:22,230-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler4) [149c59d8] HostName = virt4 2017-10-01 16:20:22,230-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler4) [149c59d8] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = virt4, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846'})' execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM 2017-10-01 16:20:22,284-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [149c59d8] START, ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 4035e3ec 2017-10-01 16:20:22,285-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [149c59d8] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:20:22,394-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [149c59d8] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:20:22,394-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [149c59d8] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:20:22,394-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [149c59d8] HostName = virt3 2017-10-01 16:20:22,394-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [149c59d8] Command 'ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:20:22,394-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [149c59d8] FINISH, ConnectStoragePoolVDSCommand, log id: 4035e3ec 2017-10-01 16:20:22,394-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler4) [149c59d8] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:20:22,480-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler4) [149c59d8] Irs placed on server 'e1e71e9d-a0cd-47db-96c6-f724f6cea736' failed. Proceed Failover 2017-10-01 16:20:22,523-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [149c59d8] START, ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 5c460829 2017-10-01 16:20:22,524-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [149c59d8] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:20:22,599-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [149c59d8] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:20:22,599-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [149c59d8] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:20:22,599-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [149c59d8] HostName = virt4 2017-10-01 16:20:22,599-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [149c59d8] Command 'ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:20:22,599-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [149c59d8] FINISH, ConnectStoragePoolVDSCommand, log id: 5c460829 2017-10-01 16:20:22,599-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler4) [149c59d8] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:20:22,690-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [149c59d8] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:20:22,690-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler4) [149c59d8] Command 'SpmStatusVDSCommand(HostName = virt4, SpmStatusVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:20:22,690-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler4) [149c59d8] Irs placed on server '5ae29ed2-9c0f-4754-8df3-b43d14726846' failed. Proceed Failover 2017-10-01 16:20:22,734-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [149c59d8] START, ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 4563d9d5 2017-10-01 16:20:22,734-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [149c59d8] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:20:22,823-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [149c59d8] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:20:22,823-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [149c59d8] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:20:22,823-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [149c59d8] HostName = virt2 2017-10-01 16:20:22,823-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [149c59d8] Command 'ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:20:22,823-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [149c59d8] FINISH, ConnectStoragePoolVDSCommand, log id: 4563d9d5 2017-10-01 16:20:22,823-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler4) [149c59d8] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:20:22,925-04 INFO [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-44) [5b10e435] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: c8653dba-63ec-4447-872c-0ffcc6af9c0f Type: Storage 2017-10-01 16:20:22,934-04 WARN [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-44) [5b10e435] skipping connect and refresh for all hosts, last master 'true', command status 'true' 2017-10-01 16:20:22,942-04 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-6-thread-41) [149c59d8] Finished reconstruct for pool 'aca61eac-ed8a-484b-bbee-3645148f6405'. Clearing event queue 2017-10-01 16:20:25,231-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler7) [242268e5] Command 'GetCapabilitiesVDSCommand(HostName = virt1, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vds='Host[virt1,45f8b331-842e-48e7-9df8-56adddb93836]'})' execution failed: java.net.NoRouteToHostException: No route to host 2017-10-01 16:20:25,231-04 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [242268e5] Failure to refresh host 'virt1' runtime info: java.net.NoRouteToHostException: No route to host 2017-10-01 16:20:28,247-04 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to virt1-mgmt/192.168.0.34 2017-10-01 16:20:30,527-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.OvfDataUpdater] (DefaultQuartzScheduler4) [5e3b5826] Attempting to update VMs/Templates Ovf. 2017-10-01 16:20:31,253-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler2) [10c6585b] Command 'GetCapabilitiesVDSCommand(HostName = virt1, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vds='Host[virt1,45f8b331-842e-48e7-9df8-56adddb93836]'})' execution failed: java.net.NoRouteToHostException: No route to host 2017-10-01 16:20:31,253-04 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [10c6585b] Failure to refresh host 'virt1' runtime info: java.net.NoRouteToHostException: No route to host 2017-10-01 16:20:33,607-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [3712ac20] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:20:33,607-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler3) [3712ac20] Command 'SpmStatusVDSCommand(HostName = virt2, SpmStatusVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:20:34,268-04 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to virt1-mgmt/192.168.0.34 2017-10-01 16:20:34,272-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [3712ac20] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command HSMGetAllTasksStatusesVDS failed: Not SPM 2017-10-01 16:20:34,272-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler3) [3712ac20] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand' return value ' 2017-10-01 16:20:34,272-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler3) [3712ac20] HostName = virt2 2017-10-01 16:20:34,272-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler3) [3712ac20] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = virt2, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e'})' execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM 2017-10-01 16:20:34,382-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [3712ac20] START, ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 132470f0 2017-10-01 16:20:34,383-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [3712ac20] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:20:34,491-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [3712ac20] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:20:34,491-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [3712ac20] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:20:34,491-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [3712ac20] HostName = virt3 2017-10-01 16:20:34,491-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [3712ac20] Command 'ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:20:34,491-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [3712ac20] FINISH, ConnectStoragePoolVDSCommand, log id: 132470f0 2017-10-01 16:20:34,491-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler3) [3712ac20] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:20:34,576-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler3) [3712ac20] Irs placed on server 'e1e71e9d-a0cd-47db-96c6-f724f6cea736' failed. Proceed Failover 2017-10-01 16:20:34,620-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [3712ac20] START, ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 7592aa2e 2017-10-01 16:20:34,620-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [3712ac20] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:20:35,183-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [3712ac20] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:20:35,183-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [3712ac20] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:20:35,183-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [3712ac20] HostName = virt4 2017-10-01 16:20:35,183-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [3712ac20] Command 'ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:20:35,183-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [3712ac20] FINISH, ConnectStoragePoolVDSCommand, log id: 7592aa2e 2017-10-01 16:20:35,183-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler3) [3712ac20] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:20:35,274-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [3712ac20] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:20:35,274-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler3) [3712ac20] Command 'SpmStatusVDSCommand(HostName = virt4, SpmStatusVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:20:35,274-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler3) [3712ac20] Irs placed on server '5ae29ed2-9c0f-4754-8df3-b43d14726846' failed. Proceed Failover 2017-10-01 16:20:35,318-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [3712ac20] START, ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 1f130744 2017-10-01 16:20:35,318-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [3712ac20] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:20:35,408-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [3712ac20] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:20:35,408-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [3712ac20] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:20:35,408-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [3712ac20] HostName = virt2 2017-10-01 16:20:35,408-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [3712ac20] Command 'ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:20:35,408-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [3712ac20] FINISH, ConnectStoragePoolVDSCommand, log id: 1f130744 2017-10-01 16:20:35,408-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler3) [3712ac20] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:20:35,510-04 INFO [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-10) [7cef0e30] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: c8653dba-63ec-4447-872c-0ffcc6af9c0f Type: Storage 2017-10-01 16:20:35,517-04 WARN [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-10) [7cef0e30] skipping connect and refresh for all hosts, last master 'true', command status 'true' 2017-10-01 16:20:35,523-04 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-6-thread-11) [3712ac20] Finished reconstruct for pool 'aca61eac-ed8a-484b-bbee-3645148f6405'. Clearing event queue 2017-10-01 16:20:37,275-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler2) [561e6b06] Command 'GetCapabilitiesVDSCommand(HostName = virt1, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vds='Host[virt1,45f8b331-842e-48e7-9df8-56adddb93836]'})' execution failed: java.net.NoRouteToHostException: No route to host 2017-10-01 16:20:37,275-04 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [561e6b06] Failure to refresh host 'virt1' runtime info: java.net.NoRouteToHostException: No route to host 2017-10-01 16:20:40,290-04 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to virt1-mgmt/192.168.0.34 2017-10-01 16:20:43,297-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler9) [3a8e4cb2] Command 'GetCapabilitiesVDSCommand(HostName = virt1, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vds='Host[virt1,45f8b331-842e-48e7-9df8-56adddb93836]'})' execution failed: java.net.NoRouteToHostException: No route to host 2017-10-01 16:20:43,297-04 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler9) [3a8e4cb2] Failure to refresh host 'virt1' runtime info: java.net.NoRouteToHostException: No route to host 2017-10-01 16:20:46,213-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [6c07866] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:20:46,213-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler8) [6c07866] Command 'SpmStatusVDSCommand(HostName = virt2, SpmStatusVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:20:46,312-04 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to virt1-mgmt/192.168.0.34 2017-10-01 16:20:46,317-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [6c07866] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command HSMGetAllTasksStatusesVDS failed: Not SPM 2017-10-01 16:20:46,317-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler8) [6c07866] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand' return value ' 2017-10-01 16:20:46,317-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler8) [6c07866] HostName = virt2 2017-10-01 16:20:46,317-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler8) [6c07866] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = virt2, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e'})' execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM 2017-10-01 16:20:46,366-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [6c07866] START, ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 3410492b 2017-10-01 16:20:46,366-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [6c07866] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:20:46,474-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [6c07866] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:20:46,474-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [6c07866] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:20:46,474-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [6c07866] HostName = virt3 2017-10-01 16:20:46,474-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [6c07866] Command 'ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:20:46,474-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [6c07866] FINISH, ConnectStoragePoolVDSCommand, log id: 3410492b 2017-10-01 16:20:46,474-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler8) [6c07866] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:20:46,561-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler8) [6c07866] Irs placed on server 'e1e71e9d-a0cd-47db-96c6-f724f6cea736' failed. Proceed Failover 2017-10-01 16:20:46,607-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [6c07866] START, ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 71f7acb7 2017-10-01 16:20:46,607-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [6c07866] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:20:46,708-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [6c07866] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:20:46,708-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [6c07866] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:20:46,708-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [6c07866] HostName = virt2 2017-10-01 16:20:46,708-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [6c07866] Command 'ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:20:46,708-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [6c07866] FINISH, ConnectStoragePoolVDSCommand, log id: 71f7acb7 2017-10-01 16:20:46,708-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler8) [6c07866] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:20:46,802-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [6c07866] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:20:46,802-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler8) [6c07866] Command 'SpmStatusVDSCommand(HostName = virt2, SpmStatusVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:20:46,802-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler8) [6c07866] Irs placed on server 'a0ccfd11-3402-4eba-a119-4987561b738e' failed. Proceed Failover 2017-10-01 16:20:46,846-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [6c07866] START, ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 353d3ea2 2017-10-01 16:20:46,846-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [6c07866] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:20:46,923-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [6c07866] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:20:46,923-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [6c07866] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:20:46,923-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [6c07866] HostName = virt4 2017-10-01 16:20:46,923-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [6c07866] Command 'ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:20:46,923-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [6c07866] FINISH, ConnectStoragePoolVDSCommand, log id: 353d3ea2 2017-10-01 16:20:46,923-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler8) [6c07866] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:20:47,020-04 INFO [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-20) [7f0fbc10] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: c8653dba-63ec-4447-872c-0ffcc6af9c0f Type: Storage 2017-10-01 16:20:47,033-04 WARN [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-20) [7f0fbc10] skipping connect and refresh for all hosts, last master 'true', command status 'true' 2017-10-01 16:20:47,039-04 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-6-thread-12) [6c07866] Finished reconstruct for pool 'aca61eac-ed8a-484b-bbee-3645148f6405'. Clearing event queue 2017-10-01 16:20:49,319-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler3) [6e7f579] Command 'GetCapabilitiesVDSCommand(HostName = virt1, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vds='Host[virt1,45f8b331-842e-48e7-9df8-56adddb93836]'})' execution failed: java.net.NoRouteToHostException: No route to host 2017-10-01 16:20:49,319-04 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [6e7f579] Failure to refresh host 'virt1' runtime info: java.net.NoRouteToHostException: No route to host 2017-10-01 16:20:52,333-04 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to virt1-mgmt/192.168.0.34 2017-10-01 16:20:55,339-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler6) [800d0e5] Command 'GetCapabilitiesVDSCommand(HostName = virt1, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vds='Host[virt1,45f8b331-842e-48e7-9df8-56adddb93836]'})' execution failed: java.net.NoRouteToHostException: No route to host 2017-10-01 16:20:55,339-04 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [800d0e5] Failure to refresh host 'virt1' runtime info: java.net.NoRouteToHostException: No route to host 2017-10-01 16:20:56,676-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [45ba2325] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 16:20:58,017-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [6a10cbe5] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:20:58,017-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler1) [6a10cbe5] Command 'SpmStatusVDSCommand(HostName = virt4, SpmStatusVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:20:58,354-04 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to virt1-mgmt/192.168.0.34 2017-10-01 16:20:58,358-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [6a10cbe5] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command HSMGetAllTasksStatusesVDS failed: Not SPM 2017-10-01 16:20:58,358-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler1) [6a10cbe5] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand' return value ' 2017-10-01 16:20:58,358-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler1) [6a10cbe5] HostName = virt4 2017-10-01 16:20:58,358-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler1) [6a10cbe5] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = virt4, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846'})' execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM 2017-10-01 16:20:58,407-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [6a10cbe5] START, ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 71f287c1 2017-10-01 16:20:58,407-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [6a10cbe5] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:20:58,517-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [6a10cbe5] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:20:58,517-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [6a10cbe5] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:20:58,517-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [6a10cbe5] HostName = virt3 2017-10-01 16:20:58,517-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [6a10cbe5] Command 'ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:20:58,517-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [6a10cbe5] FINISH, ConnectStoragePoolVDSCommand, log id: 71f287c1 2017-10-01 16:20:58,517-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler1) [6a10cbe5] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:20:58,602-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler1) [6a10cbe5] Irs placed on server 'e1e71e9d-a0cd-47db-96c6-f724f6cea736' failed. Proceed Failover 2017-10-01 16:20:58,646-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [6a10cbe5] START, ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 4d8b0779 2017-10-01 16:20:58,646-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [6a10cbe5] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:20:58,721-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [6a10cbe5] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:20:58,721-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [6a10cbe5] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:20:58,721-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [6a10cbe5] HostName = virt4 2017-10-01 16:20:58,721-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [6a10cbe5] Command 'ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:20:58,721-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [6a10cbe5] FINISH, ConnectStoragePoolVDSCommand, log id: 4d8b0779 2017-10-01 16:20:58,721-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler1) [6a10cbe5] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:20:58,813-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [6a10cbe5] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:20:58,813-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler1) [6a10cbe5] Command 'SpmStatusVDSCommand(HostName = virt4, SpmStatusVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:20:58,813-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler1) [6a10cbe5] Irs placed on server '5ae29ed2-9c0f-4754-8df3-b43d14726846' failed. Proceed Failover 2017-10-01 16:20:58,857-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [6a10cbe5] START, ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 50f222b3 2017-10-01 16:20:58,857-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [6a10cbe5] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:20:58,948-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [6a10cbe5] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:20:58,948-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [6a10cbe5] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:20:58,948-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [6a10cbe5] HostName = virt2 2017-10-01 16:20:58,948-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [6a10cbe5] Command 'ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:20:58,948-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [6a10cbe5] FINISH, ConnectStoragePoolVDSCommand, log id: 50f222b3 2017-10-01 16:20:58,948-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler1) [6a10cbe5] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:20:59,051-04 INFO [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-39) [2dbe67f7] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: c8653dba-63ec-4447-872c-0ffcc6af9c0f Type: Storage 2017-10-01 16:20:59,124-04 WARN [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-39) [2dbe67f7] skipping connect and refresh for all hosts, last master 'true', command status 'true' 2017-10-01 16:20:59,134-04 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-6-thread-22) [6a10cbe5] Finished reconstruct for pool 'aca61eac-ed8a-484b-bbee-3645148f6405'. Clearing event queue 2017-10-01 16:21:01,359-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler6) [18a9ca9f] Command 'GetCapabilitiesVDSCommand(HostName = virt1, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vds='Host[virt1,45f8b331-842e-48e7-9df8-56adddb93836]'})' execution failed: java.net.NoRouteToHostException: No route to host 2017-10-01 16:21:01,359-04 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [18a9ca9f] Failure to refresh host 'virt1' runtime info: java.net.NoRouteToHostException: No route to host 2017-10-01 16:21:04,374-04 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to virt1-mgmt/192.168.0.34 2017-10-01 16:21:07,379-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler5) [2b783255] Command 'GetCapabilitiesVDSCommand(HostName = virt1, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vds='Host[virt1,45f8b331-842e-48e7-9df8-56adddb93836]'})' execution failed: java.net.NoRouteToHostException: No route to host 2017-10-01 16:21:07,379-04 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler5) [2b783255] Failure to refresh host 'virt1' runtime info: java.net.NoRouteToHostException: No route to host 2017-10-01 16:21:09,193-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [78b257f] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:21:09,193-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler4) [78b257f] Command 'SpmStatusVDSCommand(HostName = virt2, SpmStatusVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:21:10,192-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [78b257f] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command HSMGetAllTasksStatusesVDS failed: Not SPM 2017-10-01 16:21:10,193-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler4) [78b257f] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand' return value ' 2017-10-01 16:21:10,193-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler4) [78b257f] HostName = virt2 2017-10-01 16:21:10,193-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler4) [78b257f] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = virt2, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e'})' execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM 2017-10-01 16:21:10,242-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [78b257f] START, ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 762f0df0 2017-10-01 16:21:10,242-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [78b257f] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:21:10,394-04 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to virt1-mgmt/192.168.0.34 2017-10-01 16:21:10,399-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [78b257f] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:21:10,399-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [78b257f] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:21:10,399-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [78b257f] HostName = virt4 2017-10-01 16:21:10,399-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [78b257f] Command 'ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:21:10,399-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [78b257f] FINISH, ConnectStoragePoolVDSCommand, log id: 762f0df0 2017-10-01 16:21:10,399-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler4) [78b257f] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:21:10,491-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [78b257f] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:21:10,491-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler4) [78b257f] Command 'SpmStatusVDSCommand(HostName = virt4, SpmStatusVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:21:10,491-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler4) [78b257f] Irs placed on server '5ae29ed2-9c0f-4754-8df3-b43d14726846' failed. Proceed Failover 2017-10-01 16:21:10,535-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [78b257f] START, ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 48c2e5e4 2017-10-01 16:21:10,535-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [78b257f] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:21:10,624-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [78b257f] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:21:10,624-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [78b257f] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:21:10,624-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [78b257f] HostName = virt2 2017-10-01 16:21:10,624-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [78b257f] Command 'ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:21:10,624-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [78b257f] FINISH, ConnectStoragePoolVDSCommand, log id: 48c2e5e4 2017-10-01 16:21:10,624-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler4) [78b257f] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:21:10,716-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [78b257f] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:21:10,716-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler4) [78b257f] Command 'SpmStatusVDSCommand(HostName = virt2, SpmStatusVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:21:10,716-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler4) [78b257f] Irs placed on server 'a0ccfd11-3402-4eba-a119-4987561b738e' failed. Proceed Failover 2017-10-01 16:21:10,760-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [78b257f] START, ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 4a346f68 2017-10-01 16:21:10,760-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [78b257f] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:21:10,868-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [78b257f] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:21:10,868-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [78b257f] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:21:10,868-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [78b257f] HostName = virt3 2017-10-01 16:21:10,868-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [78b257f] Command 'ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:21:10,868-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [78b257f] FINISH, ConnectStoragePoolVDSCommand, log id: 4a346f68 2017-10-01 16:21:10,868-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler4) [78b257f] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:21:10,966-04 INFO [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-33) [6ce64ada] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: c8653dba-63ec-4447-872c-0ffcc6af9c0f Type: Storage 2017-10-01 16:21:10,975-04 WARN [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-33) [6ce64ada] skipping connect and refresh for all hosts, last master 'true', command status 'true' 2017-10-01 16:21:10,981-04 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-6-thread-7) [78b257f] Finished reconstruct for pool 'aca61eac-ed8a-484b-bbee-3645148f6405'. Clearing event queue 2017-10-01 16:21:13,399-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler7) [44a87efb] Command 'GetCapabilitiesVDSCommand(HostName = virt1, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vds='Host[virt1,45f8b331-842e-48e7-9df8-56adddb93836]'})' execution failed: java.net.NoRouteToHostException: No route to host 2017-10-01 16:21:13,399-04 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler7) [44a87efb] Failure to refresh host 'virt1' runtime info: java.net.NoRouteToHostException: No route to host 2017-10-01 16:21:16,414-04 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to virt1-mgmt/192.168.0.34 2017-10-01 16:21:16,415-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler6) [6c6facd6] Command 'GetCapabilitiesVDSCommand(HostName = virt1, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vds='Host[virt1,45f8b331-842e-48e7-9df8-56adddb93836]'})' execution failed: java.net.ConnectException: Connection refused 2017-10-01 16:21:16,415-04 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [6c6facd6] Failure to refresh host 'virt1' runtime info: java.net.ConnectException: Connection refused 2017-10-01 16:21:19,431-04 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to virt1-mgmt/192.168.0.34 2017-10-01 16:21:19,431-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler2) [57eaeaca] Command 'GetCapabilitiesVDSCommand(HostName = virt1, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vds='Host[virt1,45f8b331-842e-48e7-9df8-56adddb93836]'})' execution failed: java.net.ConnectException: Connection refused 2017-10-01 16:21:19,431-04 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler2) [57eaeaca] Failure to refresh host 'virt1' runtime info: java.net.ConnectException: Connection refused 2017-10-01 16:21:22,343-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [70815043] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command HSMGetAllTasksStatusesVDS failed: Not SPM: () 2017-10-01 16:21:22,343-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler4) [70815043] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand' return value ' 2017-10-01 16:21:22,343-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler4) [70815043] HostName = virt3 2017-10-01 16:21:22,343-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler4) [70815043] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = virt3, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736'})' execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () 2017-10-01 16:21:22,394-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [70815043] START, ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 26eed79 2017-10-01 16:21:22,394-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [70815043] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:21:22,446-04 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to virt1-mgmt/192.168.0.34 2017-10-01 16:21:22,446-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler3) [51eac90b] Command 'GetCapabilitiesVDSCommand(HostName = virt1, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vds='Host[virt1,45f8b331-842e-48e7-9df8-56adddb93836]'})' execution failed: java.net.ConnectException: Connection refused 2017-10-01 16:21:22,446-04 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [51eac90b] Failure to refresh host 'virt1' runtime info: java.net.ConnectException: Connection refused 2017-10-01 16:21:23,504-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [70815043] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:21:23,504-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [70815043] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:21:23,504-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [70815043] HostName = virt3 2017-10-01 16:21:23,504-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [70815043] Command 'ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:21:23,504-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [70815043] FINISH, ConnectStoragePoolVDSCommand, log id: 26eed79 2017-10-01 16:21:23,504-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler4) [70815043] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:21:24,293-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler4) [70815043] Irs placed on server 'e1e71e9d-a0cd-47db-96c6-f724f6cea736' failed. Proceed Failover 2017-10-01 16:21:24,337-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [70815043] START, ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 510d50a1 2017-10-01 16:21:24,337-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [70815043] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:21:25,333-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [70815043] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:21:25,333-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [70815043] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:21:25,333-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [70815043] HostName = virt2 2017-10-01 16:21:25,333-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [70815043] Command 'ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:21:25,333-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [70815043] FINISH, ConnectStoragePoolVDSCommand, log id: 510d50a1 2017-10-01 16:21:25,334-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler4) [70815043] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:21:25,460-04 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to virt1-mgmt/192.168.0.34 2017-10-01 16:21:25,461-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler3) [7dda5cb6] Command 'GetCapabilitiesVDSCommand(HostName = virt1, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vds='Host[virt1,45f8b331-842e-48e7-9df8-56adddb93836]'})' execution failed: java.net.ConnectException: Connection refused 2017-10-01 16:21:25,461-04 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [7dda5cb6] Failure to refresh host 'virt1' runtime info: java.net.ConnectException: Connection refused 2017-10-01 16:21:25,466-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [70815043] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:21:25,466-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler4) [70815043] Command 'SpmStatusVDSCommand(HostName = virt2, SpmStatusVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:21:25,466-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler4) [70815043] Irs placed on server 'a0ccfd11-3402-4eba-a119-4987561b738e' failed. Proceed Failover 2017-10-01 16:21:25,510-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [70815043] START, ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 15839f31 2017-10-01 16:21:25,510-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [70815043] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:21:26,586-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [70815043] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:21:26,586-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [70815043] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:21:26,586-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [70815043] HostName = virt4 2017-10-01 16:21:26,586-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [70815043] Command 'ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:21:26,586-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [70815043] FINISH, ConnectStoragePoolVDSCommand, log id: 15839f31 2017-10-01 16:21:26,586-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler4) [70815043] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:21:26,687-04 INFO [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-40) [5e4974a8] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: c8653dba-63ec-4447-872c-0ffcc6af9c0f Type: Storage 2017-10-01 16:21:26,700-04 WARN [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-40) [5e4974a8] skipping connect and refresh for all hosts, last master 'true', command status 'true' 2017-10-01 16:21:26,706-04 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-6-thread-9) [70815043] Finished reconstruct for pool 'aca61eac-ed8a-484b-bbee-3645148f6405'. Clearing event queue 2017-10-01 16:21:28,477-04 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to virt1-mgmt/192.168.0.34 2017-10-01 16:21:29,612-04 ERROR [org.ovirt.vdsm.jsonrpc.client.JsonRpcClient] (ResponseWorker) [] Not able to update response for "c4873675-c180-434d-b6e6-7471d20e71f1" 2017-10-01 16:21:30,908-04 ERROR [org.ovirt.vdsm.jsonrpc.client.JsonRpcClient] (ResponseWorker) [] Not able to update response for "6d824bf5-0a01-45ad-8757-670de053c2ac" 2017-10-01 16:21:31,322-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [26ba286e] START, GetHardwareInfoVDSCommand(HostName = virt1, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vds='Host[virt1,45f8b331-842e-48e7-9df8-56adddb93836]'}), log id: 3aea9a80 2017-10-01 16:21:31,416-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler7) [26ba286e] FINISH, GetHardwareInfoVDSCommand, log id: 3aea9a80 2017-10-01 16:21:31,516-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler7) [1a76a230] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 45f8b331-842e-48e7-9df8-56adddb93836 Type: VDS 2017-10-01 16:21:31,541-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler7) [90fe73f] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: aca61eac-ed8a-484b-bbee-3645148f6405 Type: StoragePool 2017-10-01 16:21:31,548-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [ce0180a] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: aca61eac-ed8a-484b-bbee-3645148f6405 Type: StoragePool 2017-10-01 16:21:31,561-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [ce0180a] START, ConnectStorageServerVDSCommand(HostName = virt1, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', storageType='NFS', connectionList='[StorageServerConnections:{id='5e557659-6817-40fa-9a55-7458125404a6', connection='virtfs-fs:/nfs/data', iqn='null', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='13cf1e67-9961-4a25-a41f-cc76b4ab793d', connection='virtfs-fs:/nfs/iso', iqn='null', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='417fdd51-dbcd-41d9-9a05-ef4d9f485d13', connection='virtfs-fs:/nfs/export', iqn='null', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 32d67f4f 2017-10-01 16:21:32,795-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler7) [ce0180a] FINISH, ConnectStorageServerVDSCommand, return: {5e557659-6817-40fa-9a55-7458125404a6=0, 417fdd51-dbcd-41d9-9a05-ef4d9f485d13=0, 13cf1e67-9961-4a25-a41f-cc76b4ab793d=0}, log id: 32d67f4f 2017-10-01 16:21:32,799-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler7) [ce0180a] Host 'virt1' storage connection was succeeded 2017-10-01 16:21:32,803-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] START, ConnectStoragePoolVDSCommand(HostName = virt1, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vdsId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 208aa9a4 2017-10-01 16:21:32,803-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:21:34,213-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-25) [] FINISH, ConnectStoragePoolVDSCommand, log id: 208aa9a4 2017-10-01 16:21:34,603-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [ce0180a] EVENT_ID: FENCE_OPERATION_USING_AGENT_AND_PROXY_STARTED(9,020), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Executing power management status on Host virt1 using Proxy Host virt4 and Fence Agent ipmilan:virt1-idrac. 2017-10-01 16:21:34,603-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler7) [ce0180a] START, FenceVdsVDSCommand(HostName = virt4, FenceVdsVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', targetVdsId='45f8b331-842e-48e7-9df8-56adddb93836', action='STATUS', agent='FenceAgent:{id='b92fdde5-0407-4d57-8d00-108a0cea8300', hostId='45f8b331-842e-48e7-9df8-56adddb93836', order='1', type='ipmilan', ip='virt1-idrac', port='null', user='root', password='***', encryptOptions='false', options='privlvl=OPERATOR, delay=10, lanplus=1'}', policy='null'}), log id: 28b56543 2017-10-01 16:21:35,766-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler7) [ce0180a] FINISH, FenceVdsVDSCommand, return: FenceOperationResult:{status='SUCCESS', powerStatus='ON', message=''}, log id: 28b56543 2017-10-01 16:21:35,780-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler7) [ce0180a] START, SetMOMPolicyParametersVDSCommand(HostName = virt1, MomPolicyVDSParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836'}), log id: 3e1df27d 2017-10-01 16:21:36,676-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler7) [ce0180a] FINISH, SetMOMPolicyParametersVDSCommand, log id: 3e1df27d 2017-10-01 16:21:36,696-04 INFO [org.ovirt.engine.core.bll.hostdev.RefreshHostDevicesCommand] (DefaultQuartzScheduler7) [6bb20264] Running command: RefreshHostDevicesCommand internal: true. Entities affected : ID: 45f8b331-842e-48e7-9df8-56adddb93836 Type: VDSAction group MANIPULATE_HOST with role type ADMIN 2017-10-01 16:21:36,701-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [3c7cd614] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:21:36,701-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler1) [3c7cd614] Command 'SpmStatusVDSCommand(HostName = virt4, SpmStatusVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:21:36,820-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [3c7cd614] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command HSMGetAllTasksStatusesVDS failed: Not SPM 2017-10-01 16:21:36,820-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler1) [3c7cd614] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand' return value ' 2017-10-01 16:21:36,820-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler1) [3c7cd614] HostName = virt4 2017-10-01 16:21:36,820-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler1) [3c7cd614] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = virt4, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846'})' execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM 2017-10-01 16:21:36,859-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [6bb20264] EVENT_ID: VDS_DETECTED(13), Correlation ID: 90fe73f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Status of host virt1 was set to Up. 2017-10-01 16:21:36,862-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [6bb20264] EVENT_ID: VDS_FENCE_STATUS(496), Correlation ID: 90fe73f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host virt1 power management was verified successfully. 2017-10-01 16:21:36,875-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [3c7cd614] START, ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 77087095 2017-10-01 16:21:36,875-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [3c7cd614] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:21:36,894-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler7) [5e229706] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 45f8b331-842e-48e7-9df8-56adddb93836 Type: VDS 2017-10-01 16:21:36,899-04 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler7) [5e229706] Storage Domain 'c8653dba-63ec-4447-872c-0ffcc6af9c0f:data' was reported by Host 'virt1' as Active in Pool 'aca61eac-ed8a-484b-bbee-3645148f6405', moving to active status 2017-10-01 16:21:37,401-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [3c7cd614] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:21:37,401-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [3c7cd614] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:21:37,401-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [3c7cd614] HostName = virt2 2017-10-01 16:21:37,401-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [3c7cd614] Command 'ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:21:37,401-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [3c7cd614] FINISH, ConnectStoragePoolVDSCommand, log id: 77087095 2017-10-01 16:21:37,401-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler1) [3c7cd614] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:21:38,528-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [3c7cd614] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:21:38,528-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler1) [3c7cd614] Command 'SpmStatusVDSCommand(HostName = virt2, SpmStatusVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:21:38,528-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler1) [3c7cd614] Irs placed on server 'a0ccfd11-3402-4eba-a119-4987561b738e' failed. Proceed Failover 2017-10-01 16:21:38,586-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [3c7cd614] START, ConnectStoragePoolVDSCommand(HostName = virt1, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vdsId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 605fff5c 2017-10-01 16:21:38,586-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [3c7cd614] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:21:39,453-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [3c7cd614] FINISH, ConnectStoragePoolVDSCommand, log id: 605fff5c 2017-10-01 16:21:39,522-04 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-27) [] domain 'c8653dba-63ec-4447-872c-0ffcc6af9c0f:data' in problem 'PROBLEMATIC'. vds: 'virt4' 2017-10-01 16:21:40,510-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler1) [3c7cd614] hostFromVds::selectedVds - 'virt1', spmStatus 'Free', storage pool 'EECS-Primary', storage pool version '4.1' 2017-10-01 16:21:40,511-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler1) [3c7cd614] Found spm host 'e1e71e9d-a0cd-47db-96c6-f724f6cea736', host name: 'virt3', according to spmId: '2'. 2017-10-01 16:21:40,519-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler1) [3c7cd614] SpmStatus on vds 'e1e71e9d-a0cd-47db-96c6-f724f6cea736': 'Unknown_Pool' 2017-10-01 16:21:40,521-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler1) [3c7cd614] starting spm on vds 'virt1', storage pool 'EECS-Primary', prevId '2', LVER '60' 2017-10-01 16:21:40,521-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler1) [3c7cd614] START, SpmStartVDSCommand(HostName = virt1, SpmStartVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', prevId='2', prevLVER='60', storagePoolFormatType='V4', recoveryMode='Manual', SCSIFencing='false'}), log id: 10adb4d5 2017-10-01 16:21:41,526-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler1) [3c7cd614] spmStart polling started: taskId '500e7728-5f9c-40ec-bffc-f82ec5a4b1a9' 2017-10-01 16:21:56,696-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [27b6f4d] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 16:22:21,895-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-5) [] starting processDomainRecovery for domain 'f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2:export'. 2017-10-01 16:22:21,956-04 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-5) [] Storage domain 'f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2:export' is not visible to one or more hosts. Since the domain's type is 'ImportExport', hosts status will not be changed to non-operational 2017-10-01 16:22:21,956-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-5) [] Removing vds '[]' from the domain in maintenance cache 2017-10-01 16:22:21,957-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-5) [] Removing host(s) '[]' from hosts unseen domain report cache 2017-10-01 16:22:21,957-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-23) [] starting processDomainRecovery for domain '63d680c7-4c94-4b29-a6cb-f288a128b9bc:iso'. 2017-10-01 16:22:22,012-04 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-23) [] Storage domain '63d680c7-4c94-4b29-a6cb-f288a128b9bc:iso' is not visible to one or more hosts. Since the domain's type is 'ISO', hosts status will not be changed to non-operational 2017-10-01 16:22:22,012-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-23) [] Removing vds '[]' from the domain in maintenance cache 2017-10-01 16:22:22,012-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-23) [] Removing host(s) '[]' from hosts unseen domain report cache 2017-10-01 16:22:29,971-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] START, SetVdsStatusVDSCommand(HostName = virt1, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', status='NonResponsive', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7776e44c 2017-10-01 16:22:29,971-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] VDS 'virt1' is spm and moved from up calling resetIrs. 2017-10-01 16:22:29,971-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] START, ResetIrsVDSCommand( ResetIrsVDSCommandParameters:{runAsync='true', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', ignoreFailoverLimit='false', vdsId='45f8b331-842e-48e7-9df8-56adddb93836', ignoreStopFailed='false'}), log id: 2949f356 2017-10-01 16:22:29,985-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] START, SpmStopVDSCommand(HostName = virt1, SpmStopVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'}), log id: 723b6a4e 2017-10-01 16:22:30,633-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-47) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt1 command HSMGetAllTasksStatusesVDS failed: Not SPM 2017-10-01 16:22:30,633-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand' return value ' 2017-10-01 16:22:30,633-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] HostName = virt1 2017-10-01 16:22:30,633-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = virt1, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836'})' execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM 2017-10-01 16:22:30,633-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] FINISH, SpmStopVDSCommand, log id: 723b6a4e 2017-10-01 16:22:30,647-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] FINISH, ResetIrsVDSCommand, log id: 2949f356 2017-10-01 16:22:30,659-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-6-thread-47) [] FINISH, SetVdsStatusVDSCommand, log id: 7776e44c 2017-10-01 16:22:31,364-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [3c7cd614] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt1 command HSMGetTaskStatusVDS failed: Vds timeout occured 2017-10-01 16:22:31,364-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand] (DefaultQuartzScheduler1) [3c7cd614] Command 'HSMGetTaskStatusVDSCommand(HostName = virt1, HSMTaskGuidBaseVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', taskId='500e7728-5f9c-40ec-bffc-f82ec5a4b1a9'})' execution failed: VDSGenericException: VDSNetworkException: Vds timeout occured 2017-10-01 16:22:31,365-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler1) [3c7cd614] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand' return value 'OneUuidReturn:{status='Status [code=0, message=Done]'}' 2017-10-01 16:22:31,365-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler1) [3c7cd614] HostName = virt1 2017-10-01 16:22:31,365-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler1) [3c7cd614] Failed in 'SpmStartVDS' method, for vds: 'virt1'; host: 'virt1-mgmt': null 2017-10-01 16:22:31,365-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler1) [3c7cd614] Command 'SpmStartVDSCommand(HostName = virt1, SpmStartVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', prevId='2', prevLVER='60', storagePoolFormatType='V4', recoveryMode='Manual', SCSIFencing='false'})' execution failed: null 2017-10-01 16:22:31,365-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler1) [3c7cd614] FINISH, SpmStartVDSCommand, log id: 10adb4d5 2017-10-01 16:22:31,370-04 INFO [org.ovirt.engine.core.bll.storage.pool.SetStoragePoolStatusCommand] (DefaultQuartzScheduler1) [57dbdfe1] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: aca61eac-ed8a-484b-bbee-3645148f6405 Type: StoragePool 2017-10-01 16:22:31,371-04 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler1) [57dbdfe1] Storage Pool 'aca61eac-ed8a-484b-bbee-3645148f6405' - Updating Storage Domain '63d680c7-4c94-4b29-a6cb-f288a128b9bc' status from 'Active' to 'Unknown', reason: null 2017-10-01 16:22:31,371-04 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler1) [57dbdfe1] Storage Pool 'aca61eac-ed8a-484b-bbee-3645148f6405' - Updating Storage Domain 'f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2' status from 'Active' to 'Unknown', reason: null 2017-10-01 16:22:31,372-04 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler1) [57dbdfe1] Storage Pool 'aca61eac-ed8a-484b-bbee-3645148f6405' - Updating Storage Domain 'c8653dba-63ec-4447-872c-0ffcc6af9c0f' status from 'Active' to 'Unknown', reason: null 2017-10-01 16:22:31,381-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [57dbdfe1] EVENT_ID: SYSTEM_CHANGE_STORAGE_POOL_STATUS_PROBLEMATIC(980), Correlation ID: 57dbdfe1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Invalid status on Data Center EECS-Primary. Setting status to Non Responsive. 2017-10-01 16:22:31,381-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler1) [57dbdfe1] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: SpmStart failed 2017-10-01 16:22:31,429-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler1) [57dbdfe1] IRS failover failed - can't allocate vds server 2017-10-01 16:22:31,628-04 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to virt1-mgmt/192.168.0.34 2017-10-01 16:22:32,824-04 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-42) [] domain 'f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2:export' in problem 'NOT_REPORTED'. vds: 'virt3' 2017-10-01 16:22:32,825-04 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-42) [] domain '63d680c7-4c94-4b29-a6cb-f288a128b9bc:iso' in problem 'NOT_REPORTED'. vds: 'virt3' 2017-10-01 16:22:33,825-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [2054cd16] START, GetHardwareInfoVDSCommand(HostName = virt1, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vds='Host[virt1,45f8b331-842e-48e7-9df8-56adddb93836]'}), log id: 383059fe 2017-10-01 16:22:34,571-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler2) [2054cd16] FINISH, GetHardwareInfoVDSCommand, log id: 383059fe 2017-10-01 16:22:34,676-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler2) [1e211e2f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 45f8b331-842e-48e7-9df8-56adddb93836 Type: VDS 2017-10-01 16:22:34,703-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler2) [4fc4a669] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: aca61eac-ed8a-484b-bbee-3645148f6405 Type: StoragePool 2017-10-01 16:22:34,710-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [22508ad0] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: aca61eac-ed8a-484b-bbee-3645148f6405 Type: StoragePool 2017-10-01 16:22:34,790-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [22508ad0] START, ConnectStorageServerVDSCommand(HostName = virt1, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', storageType='NFS', connectionList='[StorageServerConnections:{id='5e557659-6817-40fa-9a55-7458125404a6', connection='virtfs-fs:/nfs/data', iqn='null', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='13cf1e67-9961-4a25-a41f-cc76b4ab793d', connection='virtfs-fs:/nfs/iso', iqn='null', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='417fdd51-dbcd-41d9-9a05-ef4d9f485d13', connection='virtfs-fs:/nfs/export', iqn='null', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 35d30a2c 2017-10-01 16:22:35,809-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler2) [22508ad0] FINISH, ConnectStorageServerVDSCommand, return: {5e557659-6817-40fa-9a55-7458125404a6=0, 417fdd51-dbcd-41d9-9a05-ef4d9f485d13=0, 13cf1e67-9961-4a25-a41f-cc76b4ab793d=0}, log id: 35d30a2c 2017-10-01 16:22:35,814-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler2) [22508ad0] Host 'virt1' storage connection was succeeded 2017-10-01 16:22:35,824-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] START, ConnectStoragePoolVDSCommand(HostName = virt1, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vdsId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 6f47f55 2017-10-01 16:22:35,824-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:22:41,586-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [6c66e337] START, ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 2385f23f 2017-10-01 16:22:41,586-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [6c66e337] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:22:41,708-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [6c66e337] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:22:41,708-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [6c66e337] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:22:41,708-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [6c66e337] HostName = virt3 2017-10-01 16:22:41,708-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [6c66e337] Command 'ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:22:41,708-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [6c66e337] FINISH, ConnectStoragePoolVDSCommand, log id: 2385f23f 2017-10-01 16:22:41,708-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler4) [6c66e337] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:22:42,795-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler4) [6c66e337] Irs placed on server 'e1e71e9d-a0cd-47db-96c6-f724f6cea736' failed. Proceed Failover 2017-10-01 16:22:42,839-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [6c66e337] START, ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 6d31fd4 2017-10-01 16:22:42,840-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [6c66e337] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:22:43,714-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [6c66e337] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:22:43,714-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [6c66e337] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:22:43,714-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [6c66e337] HostName = virt4 2017-10-01 16:22:43,714-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [6c66e337] Command 'ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:22:43,714-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [6c66e337] FINISH, ConnectStoragePoolVDSCommand, log id: 6d31fd4 2017-10-01 16:22:43,714-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler4) [6c66e337] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:22:44,805-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [6c66e337] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:22:44,806-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler4) [6c66e337] Command 'SpmStatusVDSCommand(HostName = virt4, SpmStatusVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:22:44,806-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler4) [6c66e337] Irs placed on server '5ae29ed2-9c0f-4754-8df3-b43d14726846' failed. Proceed Failover 2017-10-01 16:22:44,856-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [6c66e337] START, ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 60bae46e 2017-10-01 16:22:44,856-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [6c66e337] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:22:45,951-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [6c66e337] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:22:45,951-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [6c66e337] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:22:45,951-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [6c66e337] HostName = virt2 2017-10-01 16:22:45,951-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [6c66e337] Command 'ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:22:45,951-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler4) [6c66e337] FINISH, ConnectStoragePoolVDSCommand, log id: 60bae46e 2017-10-01 16:22:45,951-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler4) [6c66e337] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:22:56,721-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [e6d9cda] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 16:22:57,052-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [3f987fb1] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:22:57,052-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler7) [3f987fb1] Command 'SpmStatusVDSCommand(HostName = virt2, SpmStatusVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:22:58,061-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [3f987fb1] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command HSMGetAllTasksStatusesVDS failed: Not SPM 2017-10-01 16:22:58,061-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler7) [3f987fb1] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand' return value ' 2017-10-01 16:22:58,061-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler7) [3f987fb1] HostName = virt2 2017-10-01 16:22:58,061-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler7) [3f987fb1] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = virt2, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e'})' execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM 2017-10-01 16:22:58,110-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [3f987fb1] START, ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 51bbdcee 2017-10-01 16:22:58,110-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [3f987fb1] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:22:59,187-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [3f987fb1] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:22:59,187-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [3f987fb1] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:22:59,187-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [3f987fb1] HostName = virt4 2017-10-01 16:22:59,187-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [3f987fb1] Command 'ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:22:59,187-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [3f987fb1] FINISH, ConnectStoragePoolVDSCommand, log id: 51bbdcee 2017-10-01 16:22:59,187-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler7) [3f987fb1] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:23:00,279-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [3f987fb1] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:23:00,279-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler7) [3f987fb1] Command 'SpmStatusVDSCommand(HostName = virt4, SpmStatusVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:23:00,279-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler7) [3f987fb1] Irs placed on server '5ae29ed2-9c0f-4754-8df3-b43d14726846' failed. Proceed Failover 2017-10-01 16:23:00,323-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [3f987fb1] START, ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 1dfd28fc 2017-10-01 16:23:00,323-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [3f987fb1] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:23:01,505-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [3f987fb1] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:23:01,505-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [3f987fb1] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:23:01,505-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [3f987fb1] HostName = virt3 2017-10-01 16:23:01,505-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [3f987fb1] Command 'ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:23:01,505-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [3f987fb1] FINISH, ConnectStoragePoolVDSCommand, log id: 1dfd28fc 2017-10-01 16:23:01,505-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler7) [3f987fb1] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:23:02,592-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler7) [3f987fb1] Irs placed on server 'e1e71e9d-a0cd-47db-96c6-f724f6cea736' failed. Proceed Failover 2017-10-01 16:23:02,636-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [3f987fb1] START, ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 452ade42 2017-10-01 16:23:02,636-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [3f987fb1] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:23:02,908-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [3f987fb1] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:23:02,908-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [3f987fb1] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:23:02,908-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [3f987fb1] HostName = virt2 2017-10-01 16:23:02,908-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [3f987fb1] Command 'ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:23:02,908-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [3f987fb1] FINISH, ConnectStoragePoolVDSCommand, log id: 452ade42 2017-10-01 16:23:02,908-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler7) [3f987fb1] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:23:14,003-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [160f1427] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:23:14,003-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler5) [160f1427] Command 'SpmStatusVDSCommand(HostName = virt2, SpmStatusVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:23:15,011-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [160f1427] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command HSMGetAllTasksStatusesVDS failed: Not SPM 2017-10-01 16:23:15,011-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler5) [160f1427] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand' return value ' 2017-10-01 16:23:15,011-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler5) [160f1427] HostName = virt2 2017-10-01 16:23:15,011-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler5) [160f1427] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = virt2, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e'})' execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM 2017-10-01 16:23:15,060-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [160f1427] START, ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 75bf38fa 2017-10-01 16:23:15,060-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [160f1427] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:23:16,151-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [160f1427] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:23:16,151-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [160f1427] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:23:16,151-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [160f1427] HostName = virt2 2017-10-01 16:23:16,151-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [160f1427] Command 'ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:23:16,151-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [160f1427] FINISH, ConnectStoragePoolVDSCommand, log id: 75bf38fa 2017-10-01 16:23:16,151-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler5) [160f1427] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:23:17,247-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [160f1427] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:23:17,247-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler5) [160f1427] Command 'SpmStatusVDSCommand(HostName = virt2, SpmStatusVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:23:17,247-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler5) [160f1427] Irs placed on server 'a0ccfd11-3402-4eba-a119-4987561b738e' failed. Proceed Failover 2017-10-01 16:23:17,291-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [160f1427] START, ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 2077e26f 2017-10-01 16:23:17,291-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [160f1427] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:23:18,399-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [160f1427] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:23:18,399-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [160f1427] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:23:18,399-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [160f1427] HostName = virt3 2017-10-01 16:23:18,399-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [160f1427] Command 'ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:23:18,399-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [160f1427] FINISH, ConnectStoragePoolVDSCommand, log id: 2077e26f 2017-10-01 16:23:18,399-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler5) [160f1427] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:23:18,907-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler5) [160f1427] Irs placed on server 'e1e71e9d-a0cd-47db-96c6-f724f6cea736' failed. Proceed Failover 2017-10-01 16:23:18,951-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [160f1427] START, ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 7d04ddf4 2017-10-01 16:23:18,951-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [160f1427] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:23:20,028-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [160f1427] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:23:20,028-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [160f1427] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:23:20,028-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [160f1427] HostName = virt4 2017-10-01 16:23:20,028-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [160f1427] Command 'ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:23:20,028-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [160f1427] FINISH, ConnectStoragePoolVDSCommand, log id: 7d04ddf4 2017-10-01 16:23:20,028-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler5) [160f1427] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:23:31,123-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [15990e6] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:23:31,123-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler8) [15990e6] Command 'SpmStatusVDSCommand(HostName = virt4, SpmStatusVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:23:32,131-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [15990e6] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command HSMGetAllTasksStatusesVDS failed: Not SPM 2017-10-01 16:23:32,131-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler8) [15990e6] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand' return value ' 2017-10-01 16:23:32,131-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler8) [15990e6] HostName = virt4 2017-10-01 16:23:32,131-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler8) [15990e6] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = virt4, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846'})' execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM 2017-10-01 16:23:32,179-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [15990e6] START, ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 17833205 2017-10-01 16:23:32,179-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [15990e6] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:23:33,288-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [15990e6] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:23:33,288-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [15990e6] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:23:33,288-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [15990e6] HostName = virt3 2017-10-01 16:23:33,288-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [15990e6] Command 'ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:23:33,288-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [15990e6] FINISH, ConnectStoragePoolVDSCommand, log id: 17833205 2017-10-01 16:23:33,288-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler8) [15990e6] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:23:33,952-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler8) [15990e6] Irs placed on server 'e1e71e9d-a0cd-47db-96c6-f724f6cea736' failed. Proceed Failover 2017-10-01 16:23:34,009-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [15990e6] START, ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 271e1e57 2017-10-01 16:23:34,009-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [15990e6] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:23:35,099-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [15990e6] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:23:35,099-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [15990e6] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:23:35,099-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [15990e6] HostName = virt2 2017-10-01 16:23:35,099-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [15990e6] Command 'ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:23:35,099-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [15990e6] FINISH, ConnectStoragePoolVDSCommand, log id: 271e1e57 2017-10-01 16:23:35,099-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler8) [15990e6] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:23:36,192-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [15990e6] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:23:36,192-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler8) [15990e6] Command 'SpmStatusVDSCommand(HostName = virt2, SpmStatusVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:23:36,192-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler8) [15990e6] Irs placed on server 'a0ccfd11-3402-4eba-a119-4987561b738e' failed. Proceed Failover 2017-10-01 16:23:36,237-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [15990e6] START, ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 2db6497 2017-10-01 16:23:36,238-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [15990e6] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:23:37,316-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [15990e6] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:23:37,316-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [15990e6] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:23:37,316-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [15990e6] HostName = virt4 2017-10-01 16:23:37,316-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [15990e6] Command 'ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:23:37,316-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [15990e6] FINISH, ConnectStoragePoolVDSCommand, log id: 2db6497 2017-10-01 16:23:37,316-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler8) [15990e6] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:23:42,544-04 INFO [org.ovirt.engine.core.bll.MaintenanceNumberOfVdssCommand] (default task-19) [cffda9a9-5a95-41ec-970f-8486566222ce] Running command: MaintenanceNumberOfVdssCommand internal: false. Entities affected : ID: 45f8b331-842e-48e7-9df8-56adddb93836 Type: VDSAction group MANIPULATE_HOST with role type ADMIN 2017-10-01 16:23:42,596-04 INFO [org.ovirt.engine.core.bll.MaintenanceVdsCommand] (default task-19) [cffda9a9-5a95-41ec-970f-8486566222ce] Running command: MaintenanceVdsCommand internal: true. Entities affected : ID: 45f8b331-842e-48e7-9df8-56adddb93836 Type: VDS 2017-10-01 16:23:42,611-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (default task-19) [cffda9a9-5a95-41ec-970f-8486566222ce] START, SetVdsStatusVDSCommand(HostName = virt1, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', status='Maintenance', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 3423a22a 2017-10-01 16:23:48,418-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [40f1ef7c] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:23:48,418-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler6) [40f1ef7c] Command 'SpmStatusVDSCommand(HostName = virt4, SpmStatusVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:23:49,016-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [40f1ef7c] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command HSMGetAllTasksStatusesVDS failed: Not SPM 2017-10-01 16:23:49,016-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler6) [40f1ef7c] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand' return value ' 2017-10-01 16:23:49,016-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler6) [40f1ef7c] HostName = virt4 2017-10-01 16:23:49,016-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler6) [40f1ef7c] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = virt4, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846'})' execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM 2017-10-01 16:23:49,064-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [40f1ef7c] START, ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 2d8fe532 2017-10-01 16:23:49,064-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [40f1ef7c] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:23:50,172-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [40f1ef7c] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:23:50,172-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [40f1ef7c] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:23:50,172-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [40f1ef7c] HostName = virt3 2017-10-01 16:23:50,172-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [40f1ef7c] Command 'ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:23:50,172-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [40f1ef7c] FINISH, ConnectStoragePoolVDSCommand, log id: 2d8fe532 2017-10-01 16:23:50,172-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler6) [40f1ef7c] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:23:51,260-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler6) [40f1ef7c] Irs placed on server 'e1e71e9d-a0cd-47db-96c6-f724f6cea736' failed. Proceed Failover 2017-10-01 16:23:51,305-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [40f1ef7c] START, ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 3a538acf 2017-10-01 16:23:51,305-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [40f1ef7c] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:23:53,814-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [40f1ef7c] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:23:53,814-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [40f1ef7c] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:23:53,814-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [40f1ef7c] HostName = virt2 2017-10-01 16:23:53,814-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [40f1ef7c] Command 'ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:23:53,814-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [40f1ef7c] FINISH, ConnectStoragePoolVDSCommand, log id: 3a538acf 2017-10-01 16:23:53,814-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler6) [40f1ef7c] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:23:54,907-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [40f1ef7c] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:23:54,907-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler6) [40f1ef7c] Command 'SpmStatusVDSCommand(HostName = virt2, SpmStatusVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:23:54,907-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler6) [40f1ef7c] Irs placed on server 'a0ccfd11-3402-4eba-a119-4987561b738e' failed. Proceed Failover 2017-10-01 16:23:54,951-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [40f1ef7c] START, ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 72a9df5d 2017-10-01 16:23:54,951-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [40f1ef7c] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:23:56,027-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [40f1ef7c] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:23:56,027-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [40f1ef7c] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:23:56,027-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [40f1ef7c] HostName = virt4 2017-10-01 16:23:56,027-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [40f1ef7c] Command 'ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:23:56,027-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [40f1ef7c] FINISH, ConnectStoragePoolVDSCommand, log id: 72a9df5d 2017-10-01 16:23:56,027-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler6) [40f1ef7c] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:23:56,741-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [a81f332] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 16:24:07,129-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [58ab530f] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:24:07,129-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler10) [58ab530f] Command 'SpmStatusVDSCommand(HostName = virt4, SpmStatusVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:24:08,137-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [58ab530f] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command HSMGetAllTasksStatusesVDS failed: Not SPM 2017-10-01 16:24:08,137-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler10) [58ab530f] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand' return value ' 2017-10-01 16:24:08,137-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler10) [58ab530f] HostName = virt4 2017-10-01 16:24:08,137-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler10) [58ab530f] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = virt4, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846'})' execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM 2017-10-01 16:24:08,186-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [58ab530f] START, ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 4357a65e 2017-10-01 16:24:08,186-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [58ab530f] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:24:08,903-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [58ab530f] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:24:08,904-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [58ab530f] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:24:08,904-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [58ab530f] HostName = virt2 2017-10-01 16:24:08,904-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [58ab530f] Command 'ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:24:08,904-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [58ab530f] FINISH, ConnectStoragePoolVDSCommand, log id: 4357a65e 2017-10-01 16:24:08,904-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler10) [58ab530f] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:24:09,997-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [58ab530f] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:24:09,997-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler10) [58ab530f] Command 'SpmStatusVDSCommand(HostName = virt2, SpmStatusVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:24:09,997-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler10) [58ab530f] Irs placed on server 'a0ccfd11-3402-4eba-a119-4987561b738e' failed. Proceed Failover 2017-10-01 16:24:10,041-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [58ab530f] START, ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 75d4ee11 2017-10-01 16:24:10,042-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [58ab530f] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:24:11,150-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [58ab530f] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:24:11,150-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [58ab530f] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:24:11,150-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [58ab530f] HostName = virt3 2017-10-01 16:24:11,150-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [58ab530f] Command 'ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:24:11,150-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [58ab530f] FINISH, ConnectStoragePoolVDSCommand, log id: 75d4ee11 2017-10-01 16:24:11,150-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler10) [58ab530f] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:24:12,237-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler10) [58ab530f] Irs placed on server 'e1e71e9d-a0cd-47db-96c6-f724f6cea736' failed. Proceed Failover 2017-10-01 16:24:12,280-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [58ab530f] START, ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 529edc84 2017-10-01 16:24:12,281-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [58ab530f] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:24:13,357-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [58ab530f] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:24:13,357-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [58ab530f] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:24:13,357-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [58ab530f] HostName = virt4 2017-10-01 16:24:13,357-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [58ab530f] Command 'ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:24:13,357-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler10) [58ab530f] FINISH, ConnectStoragePoolVDSCommand, log id: 529edc84 2017-10-01 16:24:13,357-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler10) [58ab530f] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:24:23,994-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [656fe0e] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:24:23,994-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler7) [656fe0e] Command 'SpmStatusVDSCommand(HostName = virt4, SpmStatusVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:24:24,713-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [656fe0e] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command HSMGetAllTasksStatusesVDS failed: Not SPM 2017-10-01 16:24:24,713-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler7) [656fe0e] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand' return value ' 2017-10-01 16:24:24,713-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler7) [656fe0e] HostName = virt4 2017-10-01 16:24:24,713-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler7) [656fe0e] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = virt4, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846'})' execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM 2017-10-01 16:24:24,768-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [656fe0e] START, ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 7da4a644 2017-10-01 16:24:24,769-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [656fe0e] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:24:24,769-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-20) [] FINISH, ConnectStoragePoolVDSCommand, log id: 6f47f55 2017-10-01 16:24:24,920-04 INFO [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-38) [38523b75] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: c8653dba-63ec-4447-872c-0ffcc6af9c0f Type: Storage 2017-10-01 16:24:24,931-04 WARN [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-38) [38523b75] skipping connect and refresh for all hosts, last master 'true', command status 'true' 2017-10-01 16:24:24,939-04 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-6-thread-16) [22508ad0] Finished reconstruct for pool 'aca61eac-ed8a-484b-bbee-3645148f6405'. Clearing event queue 2017-10-01 16:24:24,939-04 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-6-thread-16) [22508ad0] The following operation 'Event:{eventType='DOMAINMONITORING', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', domainId='null', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', description=''}' was cancelled, because of reconstruct was run before 2017-10-01 16:24:24,940-04 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-6-thread-16) [22508ad0] The following operation 'Event:{eventType='RECONSTRUCT', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', domainId='c8653dba-63ec-4447-872c-0ffcc6af9c0f', vdsId='null', description='IrsBrokerCommand.startReconstruct()'}' was cancelled, because of reconstruct was run before 2017-10-01 16:24:24,940-04 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-6-thread-16) [22508ad0] The following operation 'Event:{eventType='RECONSTRUCT', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', domainId='c8653dba-63ec-4447-872c-0ffcc6af9c0f', vdsId='null', description='IrsBrokerCommand.startReconstruct()'}' was cancelled, because of reconstruct was run before 2017-10-01 16:24:24,940-04 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-6-thread-16) [22508ad0] The following operation 'Event:{eventType='RECONSTRUCT', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', domainId='c8653dba-63ec-4447-872c-0ffcc6af9c0f', vdsId='null', description='IrsBrokerCommand.startReconstruct()'}' was cancelled, because of reconstruct was run before 2017-10-01 16:24:24,940-04 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-6-thread-16) [22508ad0] The following operation 'Event:{eventType='RECONSTRUCT', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', domainId='c8653dba-63ec-4447-872c-0ffcc6af9c0f', vdsId='null', description='IrsBrokerCommand.startReconstruct()'}' was cancelled, because of reconstruct was run before 2017-10-01 16:24:24,940-04 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-6-thread-16) [22508ad0] The following operation 'Event:{eventType='RECONSTRUCT', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', domainId='c8653dba-63ec-4447-872c-0ffcc6af9c0f', vdsId='null', description='IrsBrokerCommand.startReconstruct()'}' was cancelled, because of reconstruct was run before 2017-10-01 16:24:25,049-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [22508ad0] EVENT_ID: FENCE_OPERATION_USING_AGENT_AND_PROXY_STARTED(9,020), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Executing power management status on Host virt1 using Proxy Host virt4 and Fence Agent ipmilan:virt1-idrac. 2017-10-01 16:24:25,049-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler2) [22508ad0] START, FenceVdsVDSCommand(HostName = virt4, FenceVdsVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', targetVdsId='45f8b331-842e-48e7-9df8-56adddb93836', action='STATUS', agent='FenceAgent:{id='b92fdde5-0407-4d57-8d00-108a0cea8300', hostId='45f8b331-842e-48e7-9df8-56adddb93836', order='1', type='ipmilan', ip='virt1-idrac', port='null', user='root', password='***', encryptOptions='false', options='privlvl=OPERATOR, delay=10, lanplus=1'}', policy='null'}), log id: 102a40da 2017-10-01 16:24:25,054-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [656fe0e] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:24:25,054-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [656fe0e] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:24:25,054-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [656fe0e] HostName = virt2 2017-10-01 16:24:25,054-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [656fe0e] Command 'ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:24:25,054-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [656fe0e] FINISH, ConnectStoragePoolVDSCommand, log id: 7da4a644 2017-10-01 16:24:25,054-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler7) [656fe0e] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:24:25,163-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [656fe0e] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:24:25,163-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler7) [656fe0e] Command 'SpmStatusVDSCommand(HostName = virt2, SpmStatusVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:24:25,163-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler7) [656fe0e] Irs placed on server 'a0ccfd11-3402-4eba-a119-4987561b738e' failed. Proceed Failover 2017-10-01 16:24:25,207-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [656fe0e] START, ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 53539165 2017-10-01 16:24:25,207-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [656fe0e] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:24:25,207-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler2) [22508ad0] FINISH, FenceVdsVDSCommand, return: FenceOperationResult:{status='SUCCESS', powerStatus='ON', message=''}, log id: 102a40da 2017-10-01 16:24:25,222-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler2) [22508ad0] START, SetMOMPolicyParametersVDSCommand(HostName = virt1, MomPolicyVDSParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836'}), log id: 5c4c9adb 2017-10-01 16:24:25,309-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler2) [22508ad0] FINISH, SetMOMPolicyParametersVDSCommand, log id: 5c4c9adb 2017-10-01 16:24:25,329-04 INFO [org.ovirt.engine.core.bll.hostdev.RefreshHostDevicesCommand] (DefaultQuartzScheduler2) [5257e33c] Running command: RefreshHostDevicesCommand internal: true. Entities affected : ID: 45f8b331-842e-48e7-9df8-56adddb93836 Type: VDSAction group MANIPULATE_HOST with role type ADMIN 2017-10-01 16:24:25,334-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [656fe0e] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:24:25,334-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [656fe0e] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:24:25,334-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [656fe0e] HostName = virt3 2017-10-01 16:24:25,334-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [656fe0e] Command 'ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:24:25,334-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [656fe0e] FINISH, ConnectStoragePoolVDSCommand, log id: 53539165 2017-10-01 16:24:25,334-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler7) [656fe0e] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:24:25,398-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [5257e33c] EVENT_ID: VDS_DETECTED(13), Correlation ID: 4fc4a669, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Status of host virt1 was set to Up. 2017-10-01 16:24:25,400-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [5257e33c] EVENT_ID: VDS_FENCE_STATUS(496), Correlation ID: 4fc4a669, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host virt1 power management was verified successfully. 2017-10-01 16:24:25,456-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler2) [34a0fc13] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 45f8b331-842e-48e7-9df8-56adddb93836 Type: VDS 2017-10-01 16:24:25,460-04 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler2) [34a0fc13] Storage Domain 'c8653dba-63ec-4447-872c-0ffcc6af9c0f:data' was reported by Host 'virt1' as Active in Pool 'aca61eac-ed8a-484b-bbee-3645148f6405', moving to active status 2017-10-01 16:24:25,462-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (default task-19) [cffda9a9-5a95-41ec-970f-8486566222ce] FINISH, SetVdsStatusVDSCommand, log id: 3423a22a 2017-10-01 16:24:25,467-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-19) [cffda9a9-5a95-41ec-970f-8486566222ce] EVENT_ID: USER_VDS_MAINTENANCE_WITHOUT_REASON(620), Correlation ID: cffda9a9-5a95-41ec-970f-8486566222ce, Job ID: 1bb31e20-6e1b-4b7f-8bac-f340ecebb647, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host virt1 was switched to Maintenance mode by admin@internal. 2017-10-01 16:24:26,421-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler7) [656fe0e] Irs placed on server 'e1e71e9d-a0cd-47db-96c6-f724f6cea736' failed. Proceed Failover 2017-10-01 16:24:26,466-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [656fe0e] START, ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 6c1ce34a 2017-10-01 16:24:26,466-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [656fe0e] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:24:27,561-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [656fe0e] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:24:27,561-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [656fe0e] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:24:27,561-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [656fe0e] HostName = virt4 2017-10-01 16:24:27,561-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [656fe0e] Command 'ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:24:27,561-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [656fe0e] FINISH, ConnectStoragePoolVDSCommand, log id: 6c1ce34a 2017-10-01 16:24:27,561-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler7) [656fe0e] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:24:27,660-04 INFO [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-39) [5ea28e9d] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: c8653dba-63ec-4447-872c-0ffcc6af9c0f Type: Storage 2017-10-01 16:24:27,667-04 WARN [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-39) [5ea28e9d] skipping connect and refresh for all hosts, last master 'true', command status 'true' 2017-10-01 16:24:27,681-04 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-6-thread-4) [656fe0e] Finished reconstruct for pool 'aca61eac-ed8a-484b-bbee-3645148f6405'. Clearing event queue 2017-10-01 16:24:38,655-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [746736ad] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:24:38,655-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler1) [746736ad] Command 'SpmStatusVDSCommand(HostName = virt4, SpmStatusVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:24:39,000-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [746736ad] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command HSMGetAllTasksStatusesVDS failed: Not SPM 2017-10-01 16:24:39,000-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler1) [746736ad] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand' return value ' 2017-10-01 16:24:39,000-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler1) [746736ad] HostName = virt4 2017-10-01 16:24:39,000-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler1) [746736ad] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = virt4, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846'})' execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM 2017-10-01 16:24:39,060-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [746736ad] START, ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 322106a 2017-10-01 16:24:39,060-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [746736ad] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:24:39,949-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [746736ad] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:24:39,949-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [746736ad] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:24:39,949-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [746736ad] HostName = virt4 2017-10-01 16:24:39,949-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [746736ad] Command 'ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:24:39,949-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [746736ad] FINISH, ConnectStoragePoolVDSCommand, log id: 322106a 2017-10-01 16:24:39,949-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler1) [746736ad] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:24:39,969-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-21) [] Domain 'c8653dba-63ec-4447-872c-0ffcc6af9c0f:data' recovered from problem. vds: 'virt2' 2017-10-01 16:24:40,042-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-37) [] Domain 'c8653dba-63ec-4447-872c-0ffcc6af9c0f:data' recovered from problem. vds: 'virt4' 2017-10-01 16:24:40,055-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [746736ad] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:24:40,055-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler1) [746736ad] Command 'SpmStatusVDSCommand(HostName = virt4, SpmStatusVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:24:40,055-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler1) [746736ad] Irs placed on server '5ae29ed2-9c0f-4754-8df3-b43d14726846' failed. Proceed Failover 2017-10-01 16:24:40,105-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [746736ad] START, ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 28f1eaa8 2017-10-01 16:24:40,105-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [746736ad] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:24:40,115-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-6) [] Domain 'c8653dba-63ec-4447-872c-0ffcc6af9c0f:data' recovered from problem. vds: 'virt3' 2017-10-01 16:24:40,115-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-6) [] Domain 'c8653dba-63ec-4447-872c-0ffcc6af9c0f:data' has recovered from problem. No active host in the DC is reporting it as problematic, so clearing the domain recovery timer. 2017-10-01 16:24:41,195-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [746736ad] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:24:41,195-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [746736ad] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:24:41,195-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [746736ad] HostName = virt2 2017-10-01 16:24:41,195-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [746736ad] Command 'ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:24:41,195-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [746736ad] FINISH, ConnectStoragePoolVDSCommand, log id: 28f1eaa8 2017-10-01 16:24:41,195-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler1) [746736ad] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:24:42,287-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [746736ad] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:24:42,287-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler1) [746736ad] Command 'SpmStatusVDSCommand(HostName = virt2, SpmStatusVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:24:42,288-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler1) [746736ad] Irs placed on server 'a0ccfd11-3402-4eba-a119-4987561b738e' failed. Proceed Failover 2017-10-01 16:24:42,331-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [746736ad] START, ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 3df13d2f 2017-10-01 16:24:42,331-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [746736ad] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:24:43,440-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [746736ad] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:24:43,440-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [746736ad] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:24:43,440-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [746736ad] HostName = virt3 2017-10-01 16:24:43,440-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [746736ad] Command 'ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:24:43,440-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [746736ad] FINISH, ConnectStoragePoolVDSCommand, log id: 3df13d2f 2017-10-01 16:24:43,440-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler1) [746736ad] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:24:43,546-04 INFO [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-18) [246c50a1] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: c8653dba-63ec-4447-872c-0ffcc6af9c0f Type: Storage 2017-10-01 16:24:43,553-04 WARN [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-18) [246c50a1] skipping connect and refresh for all hosts, last master 'true', command status 'true' 2017-10-01 16:24:43,566-04 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-6-thread-50) [746736ad] Finished reconstruct for pool 'aca61eac-ed8a-484b-bbee-3645148f6405'. Clearing event queue 2017-10-01 16:24:55,058-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [623a85ff] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command HSMGetAllTasksStatusesVDS failed: Not SPM: () 2017-10-01 16:24:55,058-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler6) [623a85ff] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand' return value ' 2017-10-01 16:24:55,058-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler6) [623a85ff] HostName = virt3 2017-10-01 16:24:55,058-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler6) [623a85ff] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = virt3, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736'})' execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () 2017-10-01 16:24:55,106-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [623a85ff] START, ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 2a49cdbc 2017-10-01 16:24:55,107-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [623a85ff] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:24:56,758-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [30f7862d] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 16:24:57,215-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [623a85ff] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:24:57,215-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [623a85ff] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:24:57,215-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [623a85ff] HostName = virt3 2017-10-01 16:24:57,215-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [623a85ff] Command 'ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:24:57,215-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [623a85ff] FINISH, ConnectStoragePoolVDSCommand, log id: 2a49cdbc 2017-10-01 16:24:57,215-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler6) [623a85ff] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:24:58,301-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler6) [623a85ff] Irs placed on server 'e1e71e9d-a0cd-47db-96c6-f724f6cea736' failed. Proceed Failover 2017-10-01 16:24:58,345-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [623a85ff] START, ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 62a7cfe0 2017-10-01 16:24:58,345-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [623a85ff] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:24:59,435-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [623a85ff] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:24:59,435-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [623a85ff] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:24:59,435-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [623a85ff] HostName = virt2 2017-10-01 16:24:59,435-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [623a85ff] Command 'ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:24:59,435-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [623a85ff] FINISH, ConnectStoragePoolVDSCommand, log id: 62a7cfe0 2017-10-01 16:24:59,435-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler6) [623a85ff] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:25:00,015-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [76687b4e] Autorecovering 1 storage domains 2017-10-01 16:25:00,015-04 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler2) [76687b4e] Autorecovering storage domains id: c8653dba-63ec-4447-872c-0ffcc6af9c0f 2017-10-01 16:25:00,018-04 INFO [org.ovirt.engine.core.bll.storage.connection.ConnectDomainToStorageCommand] (DefaultQuartzScheduler2) [5037a560] Running command: ConnectDomainToStorageCommand internal: true. Entities affected : ID: c8653dba-63ec-4447-872c-0ffcc6af9c0f Type: Storage 2017-10-01 16:25:00,018-04 INFO [org.ovirt.engine.core.bll.storage.connection.ConnectDomainToStorageCommand] (DefaultQuartzScheduler2) [5037a560] ConnectDomainToStorage. Before Connect all hosts to pool. Time: Sun Oct 01 16:25:00 EDT 2017 2017-10-01 16:25:00,073-04 INFO [org.ovirt.engine.core.bll.storage.connection.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-6-thread-24) [1c948f98] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN 2017-10-01 16:25:00,074-04 INFO [org.ovirt.engine.core.bll.storage.connection.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-6-thread-36) [63545072] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN 2017-10-01 16:25:00,077-04 INFO [org.ovirt.engine.core.bll.storage.connection.ConnectStorageToVdsCommand] (org.ovirt.thread.pool-6-thread-48) [671dd96d] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN 2017-10-01 16:25:00,089-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-36) [63545072] START, ConnectStorageServerVDSCommand(HostName = virt2, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='00000000-0000-0000-0000-000000000000', storageType='NFS', connectionList='[StorageServerConnections:{id='5e557659-6817-40fa-9a55-7458125404a6', connection='virtfs-fs:/nfs/data', iqn='null', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 4d8d953e 2017-10-01 16:25:00,089-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-24) [1c948f98] START, ConnectStorageServerVDSCommand(HostName = virt4, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='00000000-0000-0000-0000-000000000000', storageType='NFS', connectionList='[StorageServerConnections:{id='5e557659-6817-40fa-9a55-7458125404a6', connection='virtfs-fs:/nfs/data', iqn='null', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 12d1f302 2017-10-01 16:25:00,092-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-48) [671dd96d] START, ConnectStorageServerVDSCommand(HostName = virt3, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='00000000-0000-0000-0000-000000000000', storageType='NFS', connectionList='[StorageServerConnections:{id='5e557659-6817-40fa-9a55-7458125404a6', connection='virtfs-fs:/nfs/data', iqn='null', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 7a0a083a 2017-10-01 16:25:00,094-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [623a85ff] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt2 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:25:00,094-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler6) [623a85ff] Command 'SpmStatusVDSCommand(HostName = virt2, SpmStatusVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:25:00,094-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler6) [623a85ff] Irs placed on server 'a0ccfd11-3402-4eba-a119-4987561b738e' failed. Proceed Failover 2017-10-01 16:25:00,101-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-24) [1c948f98] FINISH, ConnectStorageServerVDSCommand, return: {5e557659-6817-40fa-9a55-7458125404a6=0}, log id: 12d1f302 2017-10-01 16:25:00,103-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-36) [63545072] FINISH, ConnectStorageServerVDSCommand, return: {5e557659-6817-40fa-9a55-7458125404a6=0}, log id: 4d8d953e 2017-10-01 16:25:00,150-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [623a85ff] START, ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 4d9f70db 2017-10-01 16:25:00,150-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [623a85ff] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:25:00,151-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (org.ovirt.thread.pool-6-thread-48) [671dd96d] FINISH, ConnectStorageServerVDSCommand, return: {5e557659-6817-40fa-9a55-7458125404a6=0}, log id: 7a0a083a 2017-10-01 16:25:00,153-04 INFO [org.ovirt.engine.core.bll.storage.connection.ConnectDomainToStorageCommand] (DefaultQuartzScheduler2) [5037a560] ConnectDomainToStorage. After Connect all hosts to pool. Time: Sun Oct 01 16:25:00 EDT 2017 2017-10-01 16:25:01,226-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [623a85ff] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:25:01,226-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [623a85ff] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:25:01,226-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [623a85ff] HostName = virt4 2017-10-01 16:25:01,226-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [623a85ff] Command 'ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:25:01,226-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [623a85ff] FINISH, ConnectStoragePoolVDSCommand, log id: 4d9f70db 2017-10-01 16:25:01,226-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler6) [623a85ff] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:25:01,336-04 INFO [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-14) [1308fb9f] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: c8653dba-63ec-4447-872c-0ffcc6af9c0f Type: Storage 2017-10-01 16:25:01,401-04 WARN [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-14) [1308fb9f] skipping connect and refresh for all hosts, last master 'true', command status 'true' 2017-10-01 16:25:01,414-04 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-6-thread-8) [623a85ff] Finished reconstruct for pool 'aca61eac-ed8a-484b-bbee-3645148f6405'. Clearing event queue 2017-10-01 16:25:12,321-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [3bd19d7d] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:25:12,321-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler5) [3bd19d7d] Command 'SpmStatusVDSCommand(HostName = virt4, SpmStatusVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:25:12,646-04 INFO [org.ovirt.engine.core.bll.MaintenanceNumberOfVdssCommand] (default task-92) [0176a198-7330-4bdc-8c1a-3266c0f815ab] Running command: MaintenanceNumberOfVdssCommand internal: false. Entities affected : ID: a0ccfd11-3402-4eba-a119-4987561b738e Type: VDSAction group MANIPULATE_HOST with role type ADMIN 2017-10-01 16:25:12,662-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (default task-92) [0176a198-7330-4bdc-8c1a-3266c0f815ab] START, SetVdsStatusVDSCommand(HostName = virt2, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', status='PreparingForMaintenance', nonOperationalReason='NONE', stopSpmFailureLogged='true', maintenanceReason='null'}), log id: 268e151b 2017-10-01 16:25:12,665-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (default task-92) [0176a198-7330-4bdc-8c1a-3266c0f815ab] FINISH, SetVdsStatusVDSCommand, log id: 268e151b 2017-10-01 16:25:12,763-04 INFO [org.ovirt.engine.core.bll.MaintenanceVdsCommand] (default task-92) [0176a198-7330-4bdc-8c1a-3266c0f815ab] Running command: MaintenanceVdsCommand internal: true. Entities affected : ID: a0ccfd11-3402-4eba-a119-4987561b738e Type: VDS 2017-10-01 16:25:12,808-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-92) [0176a198-7330-4bdc-8c1a-3266c0f815ab] EVENT_ID: USER_VDS_MAINTENANCE_WITHOUT_REASON(620), Correlation ID: 0176a198-7330-4bdc-8c1a-3266c0f815ab, Job ID: 68db7e82-e9ea-4612-823a-6bf21dc607f6, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host virt2 was switched to Maintenance mode by admin@internal. 2017-10-01 16:25:13,306-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [3bd19d7d] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command HSMGetAllTasksStatusesVDS failed: Not SPM 2017-10-01 16:25:13,306-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler5) [3bd19d7d] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand' return value ' 2017-10-01 16:25:13,306-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler5) [3bd19d7d] HostName = virt4 2017-10-01 16:25:13,306-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler5) [3bd19d7d] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = virt4, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846'})' execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM 2017-10-01 16:25:13,341-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [3bd19d7d] START, ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 4c4c72cb 2017-10-01 16:25:13,341-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [3bd19d7d] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:25:13,344-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler3) [4e252bf9] Updated host status from 'Preparing for Maintenance' to 'Maintenance' in database, host 'virt2'(a0ccfd11-3402-4eba-a119-4987561b738e) 2017-10-01 16:25:13,350-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-25) [] Clearing cache of pool: 'aca61eac-ed8a-484b-bbee-3645148f6405' for problematic entities of VDS: 'virt2'. 2017-10-01 16:25:13,350-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-25) [] Removing vds '[a0ccfd11-3402-4eba-a119-4987561b738e]' from the domain in maintenance cache 2017-10-01 16:25:13,350-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-25) [] Removing host(s) '[a0ccfd11-3402-4eba-a119-4987561b738e]' from hosts unseen domain report cache 2017-10-01 16:25:13,351-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [4e252bf9] START, DisconnectStoragePoolVDSCommand(HostName = virt2, DisconnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', vds_spm_id='3'}), log id: 79c200d 2017-10-01 16:25:13,502-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [3bd19d7d] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:25:13,502-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [3bd19d7d] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:25:13,502-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [3bd19d7d] HostName = virt3 2017-10-01 16:25:13,502-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [3bd19d7d] Command 'ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:25:13,502-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [3bd19d7d] FINISH, ConnectStoragePoolVDSCommand, log id: 4c4c72cb 2017-10-01 16:25:13,502-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler5) [3bd19d7d] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:25:13,558-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [4e252bf9] FINISH, DisconnectStoragePoolVDSCommand, log id: 79c200d 2017-10-01 16:25:13,562-04 INFO [org.ovirt.engine.core.bll.storage.pool.DisconnectHostFromStoragePoolServersCommand] (DefaultQuartzScheduler3) [5c770bd4] Running command: DisconnectHostFromStoragePoolServersCommand internal: true. Entities affected : ID: aca61eac-ed8a-484b-bbee-3645148f6405 Type: StoragePool 2017-10-01 16:25:13,568-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [5c770bd4] START, DisconnectStorageServerVDSCommand(HostName = virt2, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', storageType='NFS', connectionList='[StorageServerConnections:{id='5e557659-6817-40fa-9a55-7458125404a6', connection='virtfs-fs:/nfs/data', iqn='null', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='13cf1e67-9961-4a25-a41f-cc76b4ab793d', connection='virtfs-fs:/nfs/iso', iqn='null', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='417fdd51-dbcd-41d9-9a05-ef4d9f485d13', connection='virtfs-fs:/nfs/export', iqn='null', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 301fce47 2017-10-01 16:25:13,570-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler5) [3bd19d7d] Irs placed on server 'e1e71e9d-a0cd-47db-96c6-f724f6cea736' failed. Proceed Failover 2017-10-01 16:25:13,601-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [3bd19d7d] START, ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 352785c7 2017-10-01 16:25:13,601-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [3bd19d7d] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:25:14,682-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [3bd19d7d] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:25:14,682-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [3bd19d7d] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:25:14,682-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [3bd19d7d] HostName = virt4 2017-10-01 16:25:14,682-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [3bd19d7d] Command 'ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:25:14,682-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [3bd19d7d] FINISH, ConnectStoragePoolVDSCommand, log id: 352785c7 2017-10-01 16:25:14,682-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler5) [3bd19d7d] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:25:14,754-04 INFO [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-3) [4d94f607] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: c8653dba-63ec-4447-872c-0ffcc6af9c0f Type: Storage 2017-10-01 16:25:14,763-04 WARN [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-3) [4d94f607] skipping connect and refresh for all hosts, last master 'true', command status 'true' 2017-10-01 16:25:14,770-04 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-6-thread-26) [3bd19d7d] Finished reconstruct for pool 'aca61eac-ed8a-484b-bbee-3645148f6405'. Clearing event queue 2017-10-01 16:25:19,073-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [5c770bd4] FINISH, DisconnectStorageServerVDSCommand, return: {5e557659-6817-40fa-9a55-7458125404a6=0, 417fdd51-dbcd-41d9-9a05-ef4d9f485d13=0, 13cf1e67-9961-4a25-a41f-cc76b4ab793d=100}, log id: 301fce47 2017-10-01 16:25:22,570-04 INFO [org.ovirt.engine.core.bll.MaintenanceNumberOfVdssCommand] (default task-131) [b35a66b2-857b-4a35-b561-9e2833a778fd] Running command: MaintenanceNumberOfVdssCommand internal: false. Entities affected : ID: e1e71e9d-a0cd-47db-96c6-f724f6cea736 Type: VDSAction group MANIPULATE_HOST with role type ADMIN 2017-10-01 16:25:22,585-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (default task-131) [b35a66b2-857b-4a35-b561-9e2833a778fd] START, SetVdsStatusVDSCommand(HostName = virt3, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', status='PreparingForMaintenance', nonOperationalReason='NONE', stopSpmFailureLogged='true', maintenanceReason='null'}), log id: 5108b739 2017-10-01 16:25:22,587-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (default task-131) [b35a66b2-857b-4a35-b561-9e2833a778fd] FINISH, SetVdsStatusVDSCommand, log id: 5108b739 2017-10-01 16:25:22,688-04 INFO [org.ovirt.engine.core.bll.MaintenanceVdsCommand] (default task-131) [b35a66b2-857b-4a35-b561-9e2833a778fd] Running command: MaintenanceVdsCommand internal: true. Entities affected : ID: e1e71e9d-a0cd-47db-96c6-f724f6cea736 Type: VDS 2017-10-01 16:25:22,732-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-131) [b35a66b2-857b-4a35-b561-9e2833a778fd] EVENT_ID: USER_VDS_MAINTENANCE_WITHOUT_REASON(620), Correlation ID: b35a66b2-857b-4a35-b561-9e2833a778fd, Job ID: 82ab8d48-885b-4903-9f65-388ada1e954c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host virt3 was switched to Maintenance mode by admin@internal. 2017-10-01 16:25:24,098-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler6) [13880d46] Updated host status from 'Preparing for Maintenance' to 'Maintenance' in database, host 'virt3'(e1e71e9d-a0cd-47db-96c6-f724f6cea736) 2017-10-01 16:25:24,104-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-46) [] Clearing cache of pool: 'aca61eac-ed8a-484b-bbee-3645148f6405' for problematic entities of VDS: 'virt3'. 2017-10-01 16:25:24,104-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-46) [] Domain 'f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2:export' has recovered from problem. No active host in the DC is reporting it as poblematic, so clearing the domain recovery timer. 2017-10-01 16:25:24,105-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-46) [] Domain '63d680c7-4c94-4b29-a6cb-f288a128b9bc:iso' has recovered from problem. No active host in the DC is reporting it as poblematic, so clearing the domain recovery timer. 2017-10-01 16:25:24,105-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-46) [] Removing vds '[e1e71e9d-a0cd-47db-96c6-f724f6cea736]' from the domain in maintenance cache 2017-10-01 16:25:24,105-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-46) [] Removing host(s) '[e1e71e9d-a0cd-47db-96c6-f724f6cea736]' from hosts unseen domain report cache 2017-10-01 16:25:24,106-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [13880d46] START, DisconnectStoragePoolVDSCommand(HostName = virt3, DisconnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', vds_spm_id='2'}), log id: 132e4d78 2017-10-01 16:25:24,727-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStoragePoolVDSCommand] (DefaultQuartzScheduler6) [13880d46] FINISH, DisconnectStoragePoolVDSCommand, log id: 132e4d78 2017-10-01 16:25:24,730-04 INFO [org.ovirt.engine.core.bll.storage.pool.DisconnectHostFromStoragePoolServersCommand] (DefaultQuartzScheduler6) [5fc80b98] Running command: DisconnectHostFromStoragePoolServersCommand internal: true. Entities affected : ID: aca61eac-ed8a-484b-bbee-3645148f6405 Type: StoragePool 2017-10-01 16:25:24,745-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [5fc80b98] START, DisconnectStorageServerVDSCommand(HostName = virt3, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', storageType='NFS', connectionList='[StorageServerConnections:{id='5e557659-6817-40fa-9a55-7458125404a6', connection='virtfs-fs:/nfs/data', iqn='null', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='13cf1e67-9961-4a25-a41f-cc76b4ab793d', connection='virtfs-fs:/nfs/iso', iqn='null', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='417fdd51-dbcd-41d9-9a05-ef4d9f485d13', connection='virtfs-fs:/nfs/export', iqn='null', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: fe42fd9 2017-10-01 16:25:24,757-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [400a632f] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) 2017-10-01 16:25:24,757-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler3) [400a632f] Command 'SpmStatusVDSCommand(HostName = virt4, SpmStatusVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Error validating master storage domain: ('MD read error',) 2017-10-01 16:25:24,928-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [400a632f] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command HSMGetAllTasksStatusesVDS failed: Not SPM 2017-10-01 16:25:24,929-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler3) [400a632f] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand' return value ' 2017-10-01 16:25:24,929-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler3) [400a632f] HostName = virt4 2017-10-01 16:25:24,929-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler3) [400a632f] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = virt4, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846'})' execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM 2017-10-01 16:25:24,951-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [400a632f] START, ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 36795d1a 2017-10-01 16:25:24,951-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [400a632f] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:25:24,952-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (DefaultQuartzScheduler6) [5fc80b98] FINISH, DisconnectStorageServerVDSCommand, return: {5e557659-6817-40fa-9a55-7458125404a6=0, 417fdd51-dbcd-41d9-9a05-ef4d9f485d13=0, 13cf1e67-9961-4a25-a41f-cc76b4ab793d=0}, log id: fe42fd9 2017-10-01 16:25:25,075-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [400a632f] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:25:25,075-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [400a632f] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=304, message=Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f']]' 2017-10-01 16:25:25,075-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [400a632f] HostName = virt4 2017-10-01 16:25:25,075-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [400a632f] Command 'ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:25:25,075-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [400a632f] FINISH, ConnectStoragePoolVDSCommand, log id: 36795d1a 2017-10-01 16:25:25,075-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler3) [400a632f] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=aca61eac-ed8a-484b-bbee-3645148f6405, msdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f' 2017-10-01 16:25:25,119-04 INFO [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-11) [4c17ba03] Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: c8653dba-63ec-4447-872c-0ffcc6af9c0f Type: Storage 2017-10-01 16:25:25,130-04 WARN [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-11) [4c17ba03] skipping connect and refresh for all hosts, last master 'true', command status 'true' 2017-10-01 16:25:25,138-04 INFO [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-6-thread-27) [400a632f] Finished reconstruct for pool 'aca61eac-ed8a-484b-bbee-3645148f6405'. Clearing event queue 2017-10-01 16:25:29,384-04 INFO [org.ovirt.engine.core.bll.MaintenanceNumberOfVdssCommand] (default task-166) [89821d1a-9401-4e09-990c-5a45e100d5ae] Running command: MaintenanceNumberOfVdssCommand internal: false. Entities affected : ID: 5ae29ed2-9c0f-4754-8df3-b43d14726846 Type: VDSAction group MANIPULATE_HOST with role type ADMIN 2017-10-01 16:25:29,399-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (default task-166) [89821d1a-9401-4e09-990c-5a45e100d5ae] START, SetVdsStatusVDSCommand(HostName = virt4, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', status='PreparingForMaintenance', nonOperationalReason='NONE', stopSpmFailureLogged='true', maintenanceReason='null'}), log id: 3e25c16c 2017-10-01 16:25:29,402-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (default task-166) [89821d1a-9401-4e09-990c-5a45e100d5ae] FINISH, SetVdsStatusVDSCommand, log id: 3e25c16c 2017-10-01 16:25:29,458-04 INFO [org.ovirt.engine.core.bll.MaintenanceVdsCommand] (default task-166) [89821d1a-9401-4e09-990c-5a45e100d5ae] Running command: MaintenanceVdsCommand internal: true. Entities affected : ID: 5ae29ed2-9c0f-4754-8df3-b43d14726846 Type: VDS 2017-10-01 16:25:29,466-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-166) [89821d1a-9401-4e09-990c-5a45e100d5ae] EVENT_ID: USER_VDS_MAINTENANCE_WITHOUT_REASON(620), Correlation ID: 89821d1a-9401-4e09-990c-5a45e100d5ae, Job ID: e34cf4d2-214d-4a94-a6a4-13618939abb4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host virt4 was switched to Maintenance mode by admin@internal. 2017-10-01 16:25:30,429-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler8) [1079a9d4] Updated host status from 'Preparing for Maintenance' to 'Maintenance' in database, host 'virt4'(5ae29ed2-9c0f-4754-8df3-b43d14726846) 2017-10-01 16:25:30,435-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-47) [] Clearing cache of pool: 'aca61eac-ed8a-484b-bbee-3645148f6405' for problematic entities of VDS: 'virt4'. 2017-10-01 16:25:30,435-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-47) [] Removing vds '[5ae29ed2-9c0f-4754-8df3-b43d14726846]' from the domain in maintenance cache 2017-10-01 16:25:30,435-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (org.ovirt.thread.pool-6-thread-47) [] Removing host(s) '[5ae29ed2-9c0f-4754-8df3-b43d14726846]' from hosts unseen domain report cache 2017-10-01 16:25:30,436-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [1079a9d4] START, DisconnectStoragePoolVDSCommand(HostName = virt4, DisconnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', vds_spm_id='4'}), log id: 54802b85 2017-10-01 16:25:34,389-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStoragePoolVDSCommand] (DefaultQuartzScheduler8) [1079a9d4] FINISH, DisconnectStoragePoolVDSCommand, log id: 54802b85 2017-10-01 16:25:34,393-04 INFO [org.ovirt.engine.core.bll.storage.pool.DisconnectHostFromStoragePoolServersCommand] (DefaultQuartzScheduler8) [1f0460d2] Running command: DisconnectHostFromStoragePoolServersCommand internal: true. Entities affected : ID: aca61eac-ed8a-484b-bbee-3645148f6405 Type: StoragePool 2017-10-01 16:25:34,399-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [1f0460d2] START, DisconnectStorageServerVDSCommand(HostName = virt4, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', storageType='NFS', connectionList='[StorageServerConnections:{id='5e557659-6817-40fa-9a55-7458125404a6', connection='virtfs-fs:/nfs/data', iqn='null', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='13cf1e67-9961-4a25-a41f-cc76b4ab793d', connection='virtfs-fs:/nfs/iso', iqn='null', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='417fdd51-dbcd-41d9-9a05-ef4d9f485d13', connection='virtfs-fs:/nfs/export', iqn='null', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 21b4bde2 2017-10-01 16:25:35,077-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (DefaultQuartzScheduler8) [1f0460d2] FINISH, DisconnectStorageServerVDSCommand, return: {5e557659-6817-40fa-9a55-7458125404a6=0, 417fdd51-dbcd-41d9-9a05-ef4d9f485d13=0, 13cf1e67-9961-4a25-a41f-cc76b4ab793d=0}, log id: 21b4bde2 2017-10-01 16:25:35,092-04 INFO [org.ovirt.engine.core.vdsbroker.storage.StoragePoolDomainHelper] (DefaultQuartzScheduler3) [6ace657b] Storage Pool 'aca61eac-ed8a-484b-bbee-3645148f6405' - Updating Storage Domain 'c8653dba-63ec-4447-872c-0ffcc6af9c0f' status from 'Inactive' to 'Unknown', reason: no reporting hosts 2017-10-01 16:25:37,127-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [6ace657b] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt4 command HSMGetAllTasksStatusesVDS failed: Not SPM: () 2017-10-01 16:25:37,127-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler3) [6ace657b] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand' return value ' 2017-10-01 16:25:37,127-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler3) [6ace657b] HostName = virt4 2017-10-01 16:25:37,127-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler3) [6ace657b] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = virt4, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846'})' execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () 2017-10-01 16:25:46,368-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (default task-206) [0b7c6636-74ec-4ab4-8234-7b516d960857] Lock Acquired to object 'EngineLock:{exclusiveLocks='[45f8b331-842e-48e7-9df8-56adddb93836=VDS]', sharedLocks=''}' 2017-10-01 16:25:46,395-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-6-thread-15) [0b7c6636-74ec-4ab4-8234-7b516d960857] Running command: ActivateVdsCommand internal: false. Entities affected : ID: 45f8b331-842e-48e7-9df8-56adddb93836 Type: VDSAction group MANIPULATE_HOST with role type ADMIN 2017-10-01 16:25:46,397-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-6-thread-15) [0b7c6636-74ec-4ab4-8234-7b516d960857] Before acquiring lock in order to prevent monitoring for host 'virt1' from data-center 'EECS-Primary' 2017-10-01 16:25:46,397-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-6-thread-15) [0b7c6636-74ec-4ab4-8234-7b516d960857] Lock acquired, from now a monitoring of host will be skipped for host 'virt1' from data-center 'EECS-Primary' 2017-10-01 16:25:46,416-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-6-thread-15) [0b7c6636-74ec-4ab4-8234-7b516d960857] START, SetVdsStatusVDSCommand(HostName = virt1, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 628b0f3e 2017-10-01 16:25:46,430-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-6-thread-15) [0b7c6636-74ec-4ab4-8234-7b516d960857] FINISH, SetVdsStatusVDSCommand, log id: 628b0f3e 2017-10-01 16:25:46,433-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-6-thread-15) [0b7c6636-74ec-4ab4-8234-7b516d960857] Activate host finished. Lock released. Monitoring can run now for host 'virt1' from data-center 'EECS-Primary' 2017-10-01 16:25:46,441-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-15) [0b7c6636-74ec-4ab4-8234-7b516d960857] EVENT_ID: VDS_ACTIVATE(16), Correlation ID: 0b7c6636-74ec-4ab4-8234-7b516d960857, Job ID: 5f6704b6-a112-489c-8b45-f7b2c70c4c65, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Activation of host virt1 initiated by admin@internal. 2017-10-01 16:25:46,444-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-6-thread-15) [0b7c6636-74ec-4ab4-8234-7b516d960857] Lock freed to object 'EngineLock:{exclusiveLocks='[45f8b331-842e-48e7-9df8-56adddb93836=VDS]', sharedLocks=''}' 2017-10-01 16:25:47,219-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [1be5ff0f] START, GetHardwareInfoVDSCommand(HostName = virt1, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vds='Host[virt1,45f8b331-842e-48e7-9df8-56adddb93836]'}), log id: 78d21d0e 2017-10-01 16:25:48,223-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler1) [1be5ff0f] FINISH, GetHardwareInfoVDSCommand, log id: 78d21d0e 2017-10-01 16:25:48,328-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler1) [6fedf7ea] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 45f8b331-842e-48e7-9df8-56adddb93836 Type: VDS 2017-10-01 16:25:48,341-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler1) [3cd108b9] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: aca61eac-ed8a-484b-bbee-3645148f6405 Type: StoragePool 2017-10-01 16:25:48,347-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [fef3dea] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: aca61eac-ed8a-484b-bbee-3645148f6405 Type: StoragePool 2017-10-01 16:25:48,353-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [fef3dea] START, ConnectStorageServerVDSCommand(HostName = virt1, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', storageType='NFS', connectionList='[StorageServerConnections:{id='5e557659-6817-40fa-9a55-7458125404a6', connection='virtfs-fs:/nfs/data', iqn='null', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='13cf1e67-9961-4a25-a41f-cc76b4ab793d', connection='virtfs-fs:/nfs/iso', iqn='null', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='417fdd51-dbcd-41d9-9a05-ef4d9f485d13', connection='virtfs-fs:/nfs/export', iqn='null', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 327c9811 2017-10-01 16:25:49,371-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler1) [fef3dea] FINISH, ConnectStorageServerVDSCommand, return: {5e557659-6817-40fa-9a55-7458125404a6=0, 417fdd51-dbcd-41d9-9a05-ef4d9f485d13=0, 13cf1e67-9961-4a25-a41f-cc76b4ab793d=0}, log id: 327c9811 2017-10-01 16:25:49,375-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler1) [fef3dea] Host 'virt1' storage connection was succeeded 2017-10-01 16:25:49,379-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] START, ConnectStoragePoolVDSCommand(HostName = virt1, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vdsId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 7bdef1ad 2017-10-01 16:25:49,379-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:25:50,525-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-1) [] FINISH, ConnectStoragePoolVDSCommand, log id: 7bdef1ad 2017-10-01 16:25:51,250-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [fef3dea] EVENT_ID: FENCE_OPERATION_USING_AGENT_AND_PROXY_STARTED(9,020), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Executing power management status on Host virt1 using Proxy Host virt4 and Fence Agent ipmilan:virt1-idrac. 2017-10-01 16:25:51,250-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler1) [fef3dea] START, FenceVdsVDSCommand(HostName = virt4, FenceVdsVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', targetVdsId='45f8b331-842e-48e7-9df8-56adddb93836', action='STATUS', agent='FenceAgent:{id='b92fdde5-0407-4d57-8d00-108a0cea8300', hostId='45f8b331-842e-48e7-9df8-56adddb93836', order='1', type='ipmilan', ip='virt1-idrac', port='null', user='root', password='***', encryptOptions='false', options='privlvl=OPERATOR, delay=10, lanplus=1'}', policy='null'}), log id: 36b69134 2017-10-01 16:25:52,328-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler1) [fef3dea] FINISH, FenceVdsVDSCommand, return: FenceOperationResult:{status='SUCCESS', powerStatus='ON', message=''}, log id: 36b69134 2017-10-01 16:25:52,344-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler1) [fef3dea] START, SetMOMPolicyParametersVDSCommand(HostName = virt1, MomPolicyVDSParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836'}), log id: 15c9c0f4 2017-10-01 16:25:53,392-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler1) [fef3dea] FINISH, SetMOMPolicyParametersVDSCommand, log id: 15c9c0f4 2017-10-01 16:25:53,412-04 INFO [org.ovirt.engine.core.bll.hostdev.RefreshHostDevicesCommand] (DefaultQuartzScheduler1) [5574cfd3] Running command: RefreshHostDevicesCommand internal: true. Entities affected : ID: 45f8b331-842e-48e7-9df8-56adddb93836 Type: VDSAction group MANIPULATE_HOST with role type ADMIN 2017-10-01 16:25:53,477-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [5574cfd3] EVENT_ID: VDS_DETECTED(13), Correlation ID: 3cd108b9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Status of host virt1 was set to Up. 2017-10-01 16:25:53,480-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [5574cfd3] EVENT_ID: VDS_FENCE_STATUS(496), Correlation ID: 3cd108b9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host virt1 power management was verified successfully. 2017-10-01 16:25:53,518-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler1) [287c7e23] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 45f8b331-842e-48e7-9df8-56adddb93836 Type: VDS 2017-10-01 16:25:56,775-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [4bac94d] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 16:25:57,181-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [32287179] START, ConnectStoragePoolVDSCommand(HostName = virt1, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vdsId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 23dfd66a 2017-10-01 16:25:57,184-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [32287179] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:25:58,302-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [32287179] FINISH, ConnectStoragePoolVDSCommand, log id: 23dfd66a 2017-10-01 16:25:59,310-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler7) [32287179] hostFromVds::selectedVds - 'virt1', spmStatus 'Free', storage pool 'EECS-Primary', storage pool version '4.1' 2017-10-01 16:25:59,312-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler7) [32287179] SPM Init: could not find reported vds or not up - pool: 'EECS-Primary' vds_spm_id: '2' 2017-10-01 16:25:59,327-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler7) [32287179] SPM selection - vds seems as spm 'virt3' 2017-10-01 16:25:59,341-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (DefaultQuartzScheduler7) [32287179] START, SpmStopVDSCommand(HostName = virt3, SpmStopVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405'}), log id: 2a3e8b5f 2017-10-01 16:26:00,351-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [32287179] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VDSM virt3 command HSMGetAllTasksStatusesVDS failed: Not SPM: () 2017-10-01 16:26:00,351-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler7) [32287179] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand' return value ' 2017-10-01 16:26:00,351-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler7) [32287179] HostName = virt3 2017-10-01 16:26:00,351-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler7) [32287179] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = virt3, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736'})' execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () 2017-10-01 16:26:00,351-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (DefaultQuartzScheduler7) [32287179] FINISH, SpmStopVDSCommand, log id: 2a3e8b5f 2017-10-01 16:26:00,351-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler7) [32287179] spm stop succeeded, continuing with spm selection 2017-10-01 16:26:00,352-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler7) [32287179] starting spm on vds 'virt1', storage pool 'EECS-Primary', prevId '2', LVER '60' 2017-10-01 16:26:00,353-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler7) [32287179] START, SpmStartVDSCommand(HostName = virt1, SpmStartVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', prevId='2', prevLVER='60', storagePoolFormatType='V4', recoveryMode='Manual', SCSIFencing='true'}), log id: 61283f0c 2017-10-01 16:26:00,384-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler7) [32287179] spmStart polling started: taskId 'c1e16f51-bd55-42fe-9c43-114bf14fa070' 2017-10-01 16:26:37,466-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler7) [32287179] spmStart polling ended: taskId 'c1e16f51-bd55-42fe-9c43-114bf14fa070' task status 'finished' 2017-10-01 16:26:39,475-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler7) [32287179] spmStart polling ended, spm status: SPM 2017-10-01 16:26:39,475-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler7) [32287179] START, HSMClearTaskVDSCommand(HostName = virt1, HSMTaskGuidBaseVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', taskId='c1e16f51-bd55-42fe-9c43-114bf14fa070'}), log id: 54e252ab 2017-10-01 16:26:40,479-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler7) [32287179] FINISH, HSMClearTaskVDSCommand, log id: 54e252ab 2017-10-01 16:26:40,479-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler7) [32287179] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@32c06c7f, log id: 61283f0c 2017-10-01 16:26:40,480-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler7) [32287179] Initialize Irs proxy from vds: virt1-mgmt 2017-10-01 16:26:40,499-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [32287179] EVENT_ID: IRS_HOSTED_ON_VDS(204), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host virt1 (Address: virt1-mgmt). 2017-10-01 16:26:40,499-04 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to virt1-mgmt/192.168.0.34 2017-10-01 16:26:40,503-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-6-thread-38) [32287179] START, SPMGetAllTasksInfoVDSCommand( IrsBaseVDSCommandParameters:{runAsync='true', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', ignoreFailoverLimit='false'}), log id: 36271bea 2017-10-01 16:26:42,599-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-6-thread-38) [32287179] -- executeIrsBrokerCommand: Attempting on storage pool 'aca61eac-ed8a-484b-bbee-3645148f6405' 2017-10-01 16:26:42,599-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-6-thread-38) [32287179] START, HSMGetAllTasksInfoVDSCommand(HostName = virt1, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836'}), log id: 4e094074 2017-10-01 16:26:42,875-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-6-thread-38) [32287179] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 4e094074 2017-10-01 16:26:42,875-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (org.ovirt.thread.pool-6-thread-38) [32287179] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 36271bea 2017-10-01 16:26:42,875-04 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (org.ovirt.thread.pool-6-thread-38) [32287179] Discovered no tasks on Storage Pool 'EECS-Primary' 2017-10-01 16:26:56,795-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [26c066e4] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 16:27:56,822-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [26764948] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 16:28:15,900-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (default task-280) [56eee44e-97e2-4d50-b901-4dfb2d7efc13] Lock Acquired to object 'EngineLock:{exclusiveLocks='[a0ccfd11-3402-4eba-a119-4987561b738e=VDS]', sharedLocks=''}' 2017-10-01 16:28:15,928-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-6-thread-18) [56eee44e-97e2-4d50-b901-4dfb2d7efc13] Running command: ActivateVdsCommand internal: false. Entities affected : ID: a0ccfd11-3402-4eba-a119-4987561b738e Type: VDSAction group MANIPULATE_HOST with role type ADMIN 2017-10-01 16:28:15,928-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-6-thread-18) [56eee44e-97e2-4d50-b901-4dfb2d7efc13] Before acquiring lock in order to prevent monitoring for host 'virt2' from data-center 'EECS-Primary' 2017-10-01 16:28:15,928-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-6-thread-18) [56eee44e-97e2-4d50-b901-4dfb2d7efc13] Lock acquired, from now a monitoring of host will be skipped for host 'virt2' from data-center 'EECS-Primary' 2017-10-01 16:28:15,946-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-6-thread-18) [56eee44e-97e2-4d50-b901-4dfb2d7efc13] START, SetVdsStatusVDSCommand(HostName = virt2, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 58e7fc93 2017-10-01 16:28:15,948-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-6-thread-18) [56eee44e-97e2-4d50-b901-4dfb2d7efc13] FINISH, SetVdsStatusVDSCommand, log id: 58e7fc93 2017-10-01 16:28:15,952-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-6-thread-18) [56eee44e-97e2-4d50-b901-4dfb2d7efc13] Activate host finished. Lock released. Monitoring can run now for host 'virt2' from data-center 'EECS-Primary' 2017-10-01 16:28:15,960-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-18) [56eee44e-97e2-4d50-b901-4dfb2d7efc13] EVENT_ID: VDS_ACTIVATE(16), Correlation ID: 56eee44e-97e2-4d50-b901-4dfb2d7efc13, Job ID: dd0cc83f-42f6-40db-8ea3-92417f1cb023, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Activation of host virt2 initiated by admin@internal. 2017-10-01 16:28:15,969-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-6-thread-18) [56eee44e-97e2-4d50-b901-4dfb2d7efc13] Lock freed to object 'EngineLock:{exclusiveLocks='[a0ccfd11-3402-4eba-a119-4987561b738e=VDS]', sharedLocks=''}' 2017-10-01 16:28:17,922-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [1da38144] START, GetHardwareInfoVDSCommand(HostName = virt2, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vds='Host[virt2,a0ccfd11-3402-4eba-a119-4987561b738e]'}), log id: 7eb0e55d 2017-10-01 16:28:18,927-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler3) [1da38144] FINISH, GetHardwareInfoVDSCommand, log id: 7eb0e55d 2017-10-01 16:28:19,019-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler3) [251da998] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: a0ccfd11-3402-4eba-a119-4987561b738e Type: VDS 2017-10-01 16:28:19,047-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler3) [7b40cba0] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: aca61eac-ed8a-484b-bbee-3645148f6405 Type: StoragePool 2017-10-01 16:28:19,055-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [2a725ee1] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: aca61eac-ed8a-484b-bbee-3645148f6405 Type: StoragePool 2017-10-01 16:28:19,062-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [2a725ee1] START, ConnectStorageServerVDSCommand(HostName = virt2, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', storageType='NFS', connectionList='[StorageServerConnections:{id='5e557659-6817-40fa-9a55-7458125404a6', connection='virtfs-fs:/nfs/data', iqn='null', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='13cf1e67-9961-4a25-a41f-cc76b4ab793d', connection='virtfs-fs:/nfs/iso', iqn='null', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='417fdd51-dbcd-41d9-9a05-ef4d9f485d13', connection='virtfs-fs:/nfs/export', iqn='null', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 22c39ccb 2017-10-01 16:28:20,153-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler3) [2a725ee1] FINISH, ConnectStorageServerVDSCommand, return: {5e557659-6817-40fa-9a55-7458125404a6=0, 417fdd51-dbcd-41d9-9a05-ef4d9f485d13=0, 13cf1e67-9961-4a25-a41f-cc76b4ab793d=0}, log id: 22c39ccb 2017-10-01 16:28:20,215-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler3) [2a725ee1] Host 'virt2' storage connection was succeeded 2017-10-01 16:28:20,225-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-33) [] START, ConnectStoragePoolVDSCommand(HostName = virt2, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vdsId='a0ccfd11-3402-4eba-a119-4987561b738e', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 18429e96 2017-10-01 16:28:20,225-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-33) [] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:28:24,831-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (default task-306) [2250a3c5-cf98-425d-8129-4c0604a78273] Lock Acquired to object 'EngineLock:{exclusiveLocks='[e1e71e9d-a0cd-47db-96c6-f724f6cea736=VDS]', sharedLocks=''}' 2017-10-01 16:28:24,856-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-6-thread-35) [2250a3c5-cf98-425d-8129-4c0604a78273] Running command: ActivateVdsCommand internal: false. Entities affected : ID: e1e71e9d-a0cd-47db-96c6-f724f6cea736 Type: VDSAction group MANIPULATE_HOST with role type ADMIN 2017-10-01 16:28:24,857-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-6-thread-35) [2250a3c5-cf98-425d-8129-4c0604a78273] Before acquiring lock in order to prevent monitoring for host 'virt3' from data-center 'EECS-Primary' 2017-10-01 16:28:24,857-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-6-thread-35) [2250a3c5-cf98-425d-8129-4c0604a78273] Lock acquired, from now a monitoring of host will be skipped for host 'virt3' from data-center 'EECS-Primary' 2017-10-01 16:28:24,874-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-6-thread-35) [2250a3c5-cf98-425d-8129-4c0604a78273] START, SetVdsStatusVDSCommand(HostName = virt3, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 4b6c3da4 2017-10-01 16:28:24,877-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-6-thread-35) [2250a3c5-cf98-425d-8129-4c0604a78273] FINISH, SetVdsStatusVDSCommand, log id: 4b6c3da4 2017-10-01 16:28:24,878-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-33) [] FINISH, ConnectStoragePoolVDSCommand, log id: 18429e96 2017-10-01 16:28:24,881-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-6-thread-35) [2250a3c5-cf98-425d-8129-4c0604a78273] Activate host finished. Lock released. Monitoring can run now for host 'virt3' from data-center 'EECS-Primary' 2017-10-01 16:28:24,885-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-35) [2250a3c5-cf98-425d-8129-4c0604a78273] EVENT_ID: VDS_ACTIVATE(16), Correlation ID: 2250a3c5-cf98-425d-8129-4c0604a78273, Job ID: ebd4180c-62ad-4328-a3d2-dbb069ef746c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Activation of host virt3 initiated by admin@internal. 2017-10-01 16:28:24,889-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-6-thread-35) [2250a3c5-cf98-425d-8129-4c0604a78273] Lock freed to object 'EngineLock:{exclusiveLocks='[e1e71e9d-a0cd-47db-96c6-f724f6cea736=VDS]', sharedLocks=''}' 2017-10-01 16:28:26,062-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [2a725ee1] EVENT_ID: FENCE_OPERATION_USING_AGENT_AND_PROXY_STARTED(9,020), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Executing power management status on Host virt2 using Proxy Host virt1 and Fence Agent ipmilan:virt2-idrac. 2017-10-01 16:28:26,063-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler3) [2a725ee1] START, FenceVdsVDSCommand(HostName = virt1, FenceVdsVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', targetVdsId='a0ccfd11-3402-4eba-a119-4987561b738e', action='STATUS', agent='FenceAgent:{id='7c392b88-1692-461f-9248-12f13b0e0b01', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', order='1', type='ipmilan', ip='virt2-idrac', port='null', user='root', password='***', encryptOptions='false', options='privlvl=OPERATOR, delay=10, lanplus=1'}', policy='null'}), log id: 5fc0a4b9 2017-10-01 16:28:26,075-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [3a40abf1] START, GetHardwareInfoVDSCommand(HostName = virt3, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vds='Host[virt3,e1e71e9d-a0cd-47db-96c6-f724f6cea736]'}), log id: 687fdc2b 2017-10-01 16:28:26,294-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [3a40abf1] FINISH, GetHardwareInfoVDSCommand, log id: 687fdc2b 2017-10-01 16:28:26,386-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [6a1606cc] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: e1e71e9d-a0cd-47db-96c6-f724f6cea736 Type: VDS 2017-10-01 16:28:26,412-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [1d9ad0de] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: aca61eac-ed8a-484b-bbee-3645148f6405 Type: StoragePool 2017-10-01 16:28:26,419-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [8fc1ad8] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: aca61eac-ed8a-484b-bbee-3645148f6405 Type: StoragePool 2017-10-01 16:28:26,425-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [8fc1ad8] START, ConnectStorageServerVDSCommand(HostName = virt3, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', storageType='NFS', connectionList='[StorageServerConnections:{id='5e557659-6817-40fa-9a55-7458125404a6', connection='virtfs-fs:/nfs/data', iqn='null', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='13cf1e67-9961-4a25-a41f-cc76b4ab793d', connection='virtfs-fs:/nfs/iso', iqn='null', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='417fdd51-dbcd-41d9-9a05-ef4d9f485d13', connection='virtfs-fs:/nfs/export', iqn='null', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 7c333353 2017-10-01 16:28:26,426-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler3) [2a725ee1] FINISH, FenceVdsVDSCommand, return: FenceOperationResult:{status='SUCCESS', powerStatus='ON', message=''}, log id: 5fc0a4b9 2017-10-01 16:28:26,441-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler3) [2a725ee1] START, SetMOMPolicyParametersVDSCommand(HostName = virt2, MomPolicyVDSParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e'}), log id: 41dc3e32 2017-10-01 16:28:26,532-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler3) [2a725ee1] FINISH, SetMOMPolicyParametersVDSCommand, log id: 41dc3e32 2017-10-01 16:28:26,551-04 INFO [org.ovirt.engine.core.bll.hostdev.RefreshHostDevicesCommand] (DefaultQuartzScheduler3) [53bed74c] Running command: RefreshHostDevicesCommand internal: true. Entities affected : ID: a0ccfd11-3402-4eba-a119-4987561b738e Type: VDSAction group MANIPULATE_HOST with role type ADMIN 2017-10-01 16:28:26,553-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [8fc1ad8] FINISH, ConnectStorageServerVDSCommand, return: {5e557659-6817-40fa-9a55-7458125404a6=0, 417fdd51-dbcd-41d9-9a05-ef4d9f485d13=0, 13cf1e67-9961-4a25-a41f-cc76b4ab793d=0}, log id: 7c333353 2017-10-01 16:28:26,558-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [8fc1ad8] Host 'virt3' storage connection was succeeded 2017-10-01 16:28:26,562-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] START, ConnectStoragePoolVDSCommand(HostName = virt3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 72fbef63 2017-10-01 16:28:26,562-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:28:26,701-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [53bed74c] EVENT_ID: VDS_DETECTED(13), Correlation ID: 7b40cba0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Status of host virt2 was set to Up. 2017-10-01 16:28:26,704-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [53bed74c] EVENT_ID: VDS_FENCE_STATUS(496), Correlation ID: 7b40cba0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host virt2 power management was verified successfully. 2017-10-01 16:28:26,737-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler3) [5b660d5a] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: a0ccfd11-3402-4eba-a119-4987561b738e Type: VDS 2017-10-01 16:28:26,758-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-2) [] FINISH, ConnectStoragePoolVDSCommand, log id: 72fbef63 2017-10-01 16:28:26,907-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [8fc1ad8] EVENT_ID: FENCE_OPERATION_USING_AGENT_AND_PROXY_STARTED(9,020), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Executing power management status on Host virt3 using Proxy Host virt1 and Fence Agent ipmilan:virt3-idrac. 2017-10-01 16:28:26,907-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler10) [8fc1ad8] START, FenceVdsVDSCommand(HostName = virt1, FenceVdsVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', targetVdsId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', action='STATUS', agent='FenceAgent:{id='8c639beb-546a-4e77-9c21-1ffb54f1daae', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', order='1', type='ipmilan', ip='virt3-idrac', port='null', user='root', password='***', encryptOptions='false', options='privlvl=OPERATOR, delay=10, lanplus=1'}', policy='null'}), log id: 520b1ccb 2017-10-01 16:28:28,024-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler10) [8fc1ad8] FINISH, FenceVdsVDSCommand, return: FenceOperationResult:{status='SUCCESS', powerStatus='ON', message=''}, log id: 520b1ccb 2017-10-01 16:28:28,039-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler10) [8fc1ad8] START, SetMOMPolicyParametersVDSCommand(HostName = virt3, MomPolicyVDSParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736'}), log id: 36a0c6fe 2017-10-01 16:28:28,957-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-6-thread-48) [46f6191] START, GetFileStatsVDSCommand( GetFileStatsParameters:{runAsync='true', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', ignoreFailoverLimit='false'}), log id: 244a0f67 2017-10-01 16:28:28,957-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler10) [8fc1ad8] FINISH, SetMOMPolicyParametersVDSCommand, log id: 36a0c6fe 2017-10-01 16:28:28,976-04 INFO [org.ovirt.engine.core.bll.hostdev.RefreshHostDevicesCommand] (DefaultQuartzScheduler10) [ea47f25] Running command: RefreshHostDevicesCommand internal: true. Entities affected : ID: e1e71e9d-a0cd-47db-96c6-f724f6cea736 Type: VDSAction group MANIPULATE_HOST with role type ADMIN 2017-10-01 16:28:29,101-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-6-thread-48) [46f6191] FINISH, GetFileStatsVDSCommand, return: {virtio-win-0.1.126.iso={status=0, ctime=1497024605.0, size=155856896}, server2012r2_virtio.iso={status=0, ctime=1497297085.0, size=4977590272}, en_windows_7_professional_with_sp1_x64_dvd_621750.iso={status=0, ctime=1435288543.0, size=3319478272}, virtio-win-0.1.105.iso={status=0, ctime=1435288648.0, size=56584192}}, log id: 244a0f67 2017-10-01 16:28:29,106-04 INFO [org.ovirt.engine.core.bll.storage.domain.IsoDomainListSynchronizer] (org.ovirt.thread.pool-6-thread-48) [46f6191] Finished automatic refresh process for 'ISO' file type with success, for storage domain id '63d680c7-4c94-4b29-a6cb-f288a128b9bc'. 2017-10-01 16:28:29,137-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [ea47f25] EVENT_ID: VDS_DETECTED(13), Correlation ID: 1d9ad0de, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Status of host virt3 was set to Up. 2017-10-01 16:28:29,140-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [ea47f25] EVENT_ID: VDS_FENCE_STATUS(496), Correlation ID: 1d9ad0de, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host virt3 power management was verified successfully. 2017-10-01 16:28:29,167-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [3afcd297] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: e1e71e9d-a0cd-47db-96c6-f724f6cea736 Type: VDS 2017-10-01 16:28:30,386-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (default task-334) [a3bf4b15-e57d-4b4b-be64-1ee4b6d2fc15] Lock Acquired to object 'EngineLock:{exclusiveLocks='[5ae29ed2-9c0f-4754-8df3-b43d14726846=VDS]', sharedLocks=''}' 2017-10-01 16:28:30,414-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-6-thread-44) [a3bf4b15-e57d-4b4b-be64-1ee4b6d2fc15] Running command: ActivateVdsCommand internal: false. Entities affected : ID: 5ae29ed2-9c0f-4754-8df3-b43d14726846 Type: VDSAction group MANIPULATE_HOST with role type ADMIN 2017-10-01 16:28:30,414-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-6-thread-44) [a3bf4b15-e57d-4b4b-be64-1ee4b6d2fc15] Before acquiring lock in order to prevent monitoring for host 'virt4' from data-center 'EECS-Primary' 2017-10-01 16:28:30,414-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-6-thread-44) [a3bf4b15-e57d-4b4b-be64-1ee4b6d2fc15] Lock acquired, from now a monitoring of host will be skipped for host 'virt4' from data-center 'EECS-Primary' 2017-10-01 16:28:30,431-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-6-thread-44) [a3bf4b15-e57d-4b4b-be64-1ee4b6d2fc15] START, SetVdsStatusVDSCommand(HostName = virt4, SetVdsStatusVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 773c5483 2017-10-01 16:28:30,436-04 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (org.ovirt.thread.pool-6-thread-44) [a3bf4b15-e57d-4b4b-be64-1ee4b6d2fc15] FINISH, SetVdsStatusVDSCommand, log id: 773c5483 2017-10-01 16:28:30,441-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-6-thread-44) [a3bf4b15-e57d-4b4b-be64-1ee4b6d2fc15] Activate host finished. Lock released. Monitoring can run now for host 'virt4' from data-center 'EECS-Primary' 2017-10-01 16:28:30,451-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-44) [a3bf4b15-e57d-4b4b-be64-1ee4b6d2fc15] EVENT_ID: VDS_ACTIVATE(16), Correlation ID: a3bf4b15-e57d-4b4b-be64-1ee4b6d2fc15, Job ID: 45fdd638-4d18-445d-a98a-f4d866c7e0f4, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Activation of host virt4 initiated by admin@internal. 2017-10-01 16:28:30,454-04 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (org.ovirt.thread.pool-6-thread-44) [a3bf4b15-e57d-4b4b-be64-1ee4b6d2fc15] Lock freed to object 'EngineLock:{exclusiveLocks='[5ae29ed2-9c0f-4754-8df3-b43d14726846=VDS]', sharedLocks=''}' 2017-10-01 16:28:33,135-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [6cc8b163] START, GetHardwareInfoVDSCommand(HostName = virt4, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vds='Host[virt4,5ae29ed2-9c0f-4754-8df3-b43d14726846]'}), log id: 7f7a72cb 2017-10-01 16:28:34,138-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler10) [6cc8b163] FINISH, GetHardwareInfoVDSCommand, log id: 7f7a72cb 2017-10-01 16:28:34,241-04 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler10) [3773d7d7] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 5ae29ed2-9c0f-4754-8df3-b43d14726846 Type: VDS 2017-10-01 16:28:34,268-04 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler10) [29a7a380] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: aca61eac-ed8a-484b-bbee-3645148f6405 Type: StoragePool 2017-10-01 16:28:34,275-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [272f1d] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: aca61eac-ed8a-484b-bbee-3645148f6405 Type: StoragePool 2017-10-01 16:28:34,280-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [272f1d] START, ConnectStorageServerVDSCommand(HostName = virt4, StorageServerConnectionManagementVDSParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', storageType='NFS', connectionList='[StorageServerConnections:{id='5e557659-6817-40fa-9a55-7458125404a6', connection='virtfs-fs:/nfs/data', iqn='null', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='13cf1e67-9961-4a25-a41f-cc76b4ab793d', connection='virtfs-fs:/nfs/iso', iqn='null', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='417fdd51-dbcd-41d9-9a05-ef4d9f485d13', connection='virtfs-fs:/nfs/export', iqn='null', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]'}), log id: 492de0d0 2017-10-01 16:28:35,098-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler10) [272f1d] FINISH, ConnectStorageServerVDSCommand, return: {5e557659-6817-40fa-9a55-7458125404a6=0, 417fdd51-dbcd-41d9-9a05-ef4d9f485d13=0, 13cf1e67-9961-4a25-a41f-cc76b4ab793d=0}, log id: 492de0d0 2017-10-01 16:28:35,102-04 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler10) [272f1d] Host 'virt4' storage connection was succeeded 2017-10-01 16:28:35,112-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] START, ConnectStoragePoolVDSCommand(HostName = virt4, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', masterVersion='1'}), log id: 48ceb64b 2017-10-01 16:28:35,112-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] Executing with domain map: {c8653dba-63ec-4447-872c-0ffcc6af9c0f=active, f1eb9be5-1bff-4ffa-bfca-c8d53a176fc2=active, 63d680c7-4c94-4b29-a6cb-f288a128b9bc=active} 2017-10-01 16:28:36,268-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-6-thread-9) [] FINISH, ConnectStoragePoolVDSCommand, log id: 48ceb64b 2017-10-01 16:28:37,135-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [272f1d] EVENT_ID: FENCE_OPERATION_USING_AGENT_AND_PROXY_STARTED(9,020), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Executing power management status on Host virt4 using Proxy Host virt1 and Fence Agent ipmilan:virt4-idrac. 2017-10-01 16:28:37,135-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler10) [272f1d] START, FenceVdsVDSCommand(HostName = virt1, FenceVdsVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', targetVdsId='5ae29ed2-9c0f-4754-8df3-b43d14726846', action='STATUS', agent='FenceAgent:{id='6a1fc1bb-0358-475a-aed2-e9c1868a2fc8', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', order='1', type='ipmilan', ip='virt4-idrac', port='null', user='root', password='***', encryptOptions='false', options='privlvl=OPERATOR, delay=10, lanplus=1'}', policy='null'}), log id: 4eac4a11 2017-10-01 16:28:38,256-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FenceVdsVDSCommand] (DefaultQuartzScheduler10) [272f1d] FINISH, FenceVdsVDSCommand, return: FenceOperationResult:{status='SUCCESS', powerStatus='ON', message=''}, log id: 4eac4a11 2017-10-01 16:28:38,271-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler10) [272f1d] START, SetMOMPolicyParametersVDSCommand(HostName = virt4, MomPolicyVDSParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846'}), log id: 7146cde3 2017-10-01 16:28:39,313-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler10) [272f1d] FINISH, SetMOMPolicyParametersVDSCommand, log id: 7146cde3 2017-10-01 16:28:39,332-04 INFO [org.ovirt.engine.core.bll.hostdev.RefreshHostDevicesCommand] (DefaultQuartzScheduler10) [3322a373] Running command: RefreshHostDevicesCommand internal: true. Entities affected : ID: 5ae29ed2-9c0f-4754-8df3-b43d14726846 Type: VDSAction group MANIPULATE_HOST with role type ADMIN 2017-10-01 16:28:39,563-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [3322a373] EVENT_ID: VDS_DETECTED(13), Correlation ID: 29a7a380, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Status of host virt4 was set to Up. 2017-10-01 16:28:39,567-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [3322a373] EVENT_ID: VDS_FENCE_STATUS(496), Correlation ID: 29a7a380, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host virt4 power management was verified successfully. 2017-10-01 16:28:39,615-04 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler10) [216a2686] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5ae29ed2-9c0f-4754-8df3-b43d14726846 Type: VDS 2017-10-01 16:28:56,845-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler7) [1552e356] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 16:29:23,111-04 INFO [org.ovirt.engine.core.bll.RunVmCommand] (default task-380) [b5599ec5-85b5-418d-8d1f-58fa9db827ef] Lock Acquired to object 'EngineLock:{exclusiveLocks='[8391f9a7-7fe7-4025-8fba-6273ae587a52=VM]', sharedLocks=''}' 2017-10-01 16:29:23,137-04 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (default task-380) [b5599ec5-85b5-418d-8d1f-58fa9db827ef] START, IsVmDuringInitiatingVDSCommand( IsVmDuringInitiatingVDSCommandParameters:{runAsync='true', vmId='8391f9a7-7fe7-4025-8fba-6273ae587a52'}), log id: 19d8e469 2017-10-01 16:29:23,137-04 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (default task-380) [b5599ec5-85b5-418d-8d1f-58fa9db827ef] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 19d8e469 2017-10-01 16:29:23,234-04 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-6-thread-22) [b5599ec5-85b5-418d-8d1f-58fa9db827ef] Running command: RunVmCommand internal: false. Entities affected : ID: 8391f9a7-7fe7-4025-8fba-6273ae587a52 Type: VMAction group RUN_VM with role type USER 2017-10-01 16:29:23,352-04 INFO [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (org.ovirt.thread.pool-6-thread-22) [b5599ec5-85b5-418d-8d1f-58fa9db827ef] START, UpdateVmDynamicDataVDSCommand( UpdateVmDynamicDataVDSCommandParameters:{runAsync='true', hostId='null', vmId='8391f9a7-7fe7-4025-8fba-6273ae587a52', vmDynamic='org.ovirt.engine.core.common.businessentities.VmDynamic@129ab91e'}), log id: 2c8fbd8 2017-10-01 16:29:23,354-04 INFO [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (org.ovirt.thread.pool-6-thread-22) [b5599ec5-85b5-418d-8d1f-58fa9db827ef] FINISH, UpdateVmDynamicDataVDSCommand, log id: 2c8fbd8 2017-10-01 16:29:23,357-04 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-6-thread-22) [b5599ec5-85b5-418d-8d1f-58fa9db827ef] START, CreateVmVDSCommand( CreateVmVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vmId='8391f9a7-7fe7-4025-8fba-6273ae587a52', vm='VM [archive]'}), log id: 3422a15a 2017-10-01 16:29:23,359-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-6-thread-22) [b5599ec5-85b5-418d-8d1f-58fa9db827ef] START, CreateVDSCommand(HostName = virt3, CreateVmVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vmId='8391f9a7-7fe7-4025-8fba-6273ae587a52', vm='VM [archive]'}), log id: 62b8f4ce 2017-10-01 16:29:23,362-04 INFO [org.ovirt.engine.core.vdsbroker.builder.vminfo.VmInfoBuilderImpl] (org.ovirt.thread.pool-6-thread-22) [b5599ec5-85b5-418d-8d1f-58fa9db827ef] Bootable disk '9b65995d-f733-4a0a-aaa1-d9f116f4ccd7' set to index '0' 2017-10-01 16:29:23,377-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-6-thread-22) [b5599ec5-85b5-418d-8d1f-58fa9db827ef] VM {memGuaranteedSize=682, smpThreadsPerCore=1, cpuType=SandyBridge, vmId=8391f9a7-7fe7-4025-8fba-6273ae587a52, acpiEnable=true, vmType=kvm, smp=1, smpCoresPerSocket=1, emulatedMachine=pc-i440fx-rhel7.3.0, smartcardEnable=false, guestNumaNodes=[{memory=1024, cpus=0, nodeIndex=0}], transparentHugePages=true, displayNetwork=yorku, vmName=archive, maxVCpus=16, kvmEnable=true, devices=[{address={bus=0x00, domain=0x0000, function=0x0, slot=0x02, type=pci}, type=video, specParams={heads=1, vram=32768}, device=qxl, deviceId=51f31a4d-979a-4dba-a983-fda800d7e33d}, {type=graphics, specParams={fileTransferEnable=true, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,ssmartcard,susbredir, copyPasteEnable=true}, device=spice, deviceId=c520628b-2ea8-4b60-b6c6-590abc0e7ef8}, {iface=ide, shared=false, path=, address={bus=1, controller=0, unit=0, type=drive, target=0}, readonly=true, index=2, type=disk, specParams={path=}, device=cdrom, deviceId=12e9e84b-395e-4cb4-8b9c-5f96604351f2}, {discard=false, shared=false, address={bus=0x00, domain=0x0000, function=0x0, slot=0x07, type=pci}, imageID=9b65995d-f733-4a0a-aaa1-d9f116f4ccd7, format=raw, index=0, optional=false, type=disk, deviceId=9b65995d-f733-4a0a-aaa1-d9f116f4ccd7, domainID=c8653dba-63ec-4447-872c-0ffcc6af9c0f, propagateErrors=off, iface=virtio, readonly=false, bootOrder=1, poolID=aca61eac-ed8a-484b-bbee-3645148f6405, volumeID=bc2bd39d-e0f4-4cc8-ad39-6cb958d27f6b, specParams={}, device=disk}, {filter=vdsm-no-mac-spoofing, nicModel=pv, address={bus=0x00, domain=0x0000, function=0x0, slot=0x03, type=pci}, type=interface, specParams={inbound={}, outbound={}}, device=bridge, linkActive=true, deviceId=dd0ae5e5-1078-486f-9fc1-7604da8293fc, macAddr=00:1a:4a:32:3d:2c, network=yorku}, {filter=vdsm-no-mac-spoofing, nicModel=pv, address={bus=0x00, domain=0x0000, function=0x0, slot=0x04, type=pci}, type=interface, specParams={inbound={}, outbound={}}, device=bridge, linkActive=true, deviceId=28df81a9-1c17-429d-a192-8b5525af584f, macAddr=00:1a:4a:32:3d:2d, network=mrpriv}, {address={bus=0x00, domain=0x0000, function=0x2, slot=0x01, type=pci}, index=0, model=piix3-uhci, type=controller, specParams={}, device=usb, deviceId=626533f6-e2ac-4e0f-88ea-283d26764eeb}, {address={bus=0x00, domain=0x0000, function=0x0, slot=0x05, type=pci}, index=0, model=virtio-scsi, type=controller, specParams={}, device=scsi, deviceId=6ffb1266-df8b-4cb4-ac11-7267a2fdbe11}, {address={bus=0x00, domain=0x0000, function=0x0, slot=0x06, type=pci}, type=controller, specParams={}, device=virtio-serial, deviceId=204379a0-123e-4efd-afe4-6e56501364df}, {model=virtio, type=rng, specParams={source=urandom}, device=virtio, deviceId=93aa49d4-7f7f-43d8-9ae2-792418b96e85}], custom={device_41de5a24-f2a9-4045-87b6-83e021974d30device_90ae615b-8160-471f-ac1e-d956d1d70f73device_104d2c39-4bd9-43a8-9baa-7b99cc0784a2device_2e94969e-d2b2-4f54-aab0-f9aba03c7475=VmDevice:{id='VmDeviceId:{deviceId='2e94969e-d2b2-4f54-aab0-f9aba03c7475', vmId='8391f9a7-7fe7-4025-8fba-6273ae587a52'}', device='ide', type='CONTROLLER', bootOrder='0', specParams='[]', address='{slot=0x01, bus=0x00, domain=0x0000, type=pci, function=0x1}', managed='false', plugged='true', readOnly='false', deviceAlias='ide', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_41de5a24-f2a9-4045-87b6-83e021974d30=VmDevice:{id='VmDeviceId:{deviceId='41de5a24-f2a9-4045-87b6-83e021974d30', vmId='8391f9a7-7fe7-4025-8fba-6273ae587a52'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=1}', managed='false', plugged='true', readOnly='false', deviceAlias='channel0', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_41de5a24-f2a9-4045-87b6-83e021974d30device_90ae615b-8160-471f-ac1e-d956d1d70f73=VmDevice:{id='VmDeviceId:{deviceId='90ae615b-8160-471f-ac1e-d956d1d70f73', vmId='8391f9a7-7fe7-4025-8fba-6273ae587a52'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=2}', managed='false', plugged='true', readOnly='false', deviceAlias='channel1', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_41de5a24-f2a9-4045-87b6-83e021974d30device_90ae615b-8160-471f-ac1e-d956d1d70f73device_104d2c39-4bd9-43a8-9baa-7b99cc0784a2=VmDevice:{id='VmDeviceId:{deviceId='104d2c39-4bd9-43a8-9baa-7b99cc0784a2', vmId='8391f9a7-7fe7-4025-8fba-6273ae587a52'}', device='spicevmc', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=3}', managed='false', plugged='true', readOnly='false', deviceAlias='channel2', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}}, display=qxl, timeOffset=0, nice=0, maxMemSize=4096, maxMemSlots=16, bootMenuEnable=false, memSize=1024} 2017-10-01 16:29:23,391-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-6-thread-22) [b5599ec5-85b5-418d-8d1f-58fa9db827ef] FINISH, CreateVDSCommand, log id: 62b8f4ce 2017-10-01 16:29:23,394-04 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-6-thread-22) [b5599ec5-85b5-418d-8d1f-58fa9db827ef] FINISH, CreateVmVDSCommand, return: WaitForLaunch, log id: 3422a15a 2017-10-01 16:29:23,395-04 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-6-thread-22) [b5599ec5-85b5-418d-8d1f-58fa9db827ef] Lock freed to object 'EngineLock:{exclusiveLocks='[8391f9a7-7fe7-4025-8fba-6273ae587a52=VM]', sharedLocks=''}' 2017-10-01 16:29:23,403-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-22) [b5599ec5-85b5-418d-8d1f-58fa9db827ef] EVENT_ID: USER_STARTED_VM(153), Correlation ID: b5599ec5-85b5-418d-8d1f-58fa9db827ef, Job ID: d5b73987-8137-4850-adaa-eda2b6308572, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM archive was started by admin@internal (Host: virt3). 2017-10-01 16:29:27,969-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmsStatisticsFetcher] (DefaultQuartzScheduler7) [6b3cf907] Fetched 1 VMs from VDS 'e1e71e9d-a0cd-47db-96c6-f724f6cea736' 2017-10-01 16:29:27,969-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler7) [6b3cf907] VM '8391f9a7-7fe7-4025-8fba-6273ae587a52'(archive) moved from 'WaitForLaunch' --> 'PoweringUp' 2017-10-01 16:29:27,975-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVDSCommand] (DefaultQuartzScheduler7) [6b3cf907] START, FullListVDSCommand(HostName = virt3, FullListVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vmIds='[8391f9a7-7fe7-4025-8fba-6273ae587a52]'}), log id: 26f8a931 2017-10-01 16:29:28,162-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVDSCommand] (DefaultQuartzScheduler7) [6b3cf907] FINISH, FullListVDSCommand, return: [{acpiEnable=true, emulatedMachine=pc-i440fx-rhel7.3.0, vmId=8391f9a7-7fe7-4025-8fba-6273ae587a52, guestDiskMapping={}, transparentHugePages=true, timeOffset=0, cpuType=SandyBridge, smp=1, pauseCode=NOERR, guestNumaNodes=[Ljava.lang.Object;@1c230ee3, smartcardEnable=false, custom={device_41de5a24-f2a9-4045-87b6-83e021974d30device_90ae615b-8160-471f-ac1e-d956d1d70f73device_104d2c39-4bd9-43a8-9baa-7b99cc0784a2device_2e94969e-d2b2-4f54-aab0-f9aba03c7475=VmDevice:{id='VmDeviceId:{deviceId='2e94969e-d2b2-4f54-aab0-f9aba03c7475', vmId='8391f9a7-7fe7-4025-8fba-6273ae587a52'}', device='ide', type='CONTROLLER', bootOrder='0', specParams='[]', address='{slot=0x01, bus=0x00, domain=0x0000, type=pci, function=0x1}', managed='false', plugged='true', readOnly='false', deviceAlias='ide', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_41de5a24-f2a9-4045-87b6-83e021974d30device_90ae615b-8160-471f-ac1e-d956d1d70f73=VmDevice:{id='VmDeviceId:{deviceId='90ae615b-8160-471f-ac1e-d956d1d70f73', vmId='8391f9a7-7fe7-4025-8fba-6273ae587a52'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=2}', managed='false', plugged='true', readOnly='false', deviceAlias='channel1', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_41de5a24-f2a9-4045-87b6-83e021974d30device_90ae615b-8160-471f-ac1e-d956d1d70f73device_104d2c39-4bd9-43a8-9baa-7b99cc0784a2=VmDevice:{id='VmDeviceId:{deviceId='104d2c39-4bd9-43a8-9baa-7b99cc0784a2', vmId='8391f9a7-7fe7-4025-8fba-6273ae587a52'}', device='spicevmc', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=3}', managed='false', plugged='true', readOnly='false', deviceAlias='channel2', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_41de5a24-f2a9-4045-87b6-83e021974d30=VmDevice:{id='VmDeviceId:{deviceId='41de5a24-f2a9-4045-87b6-83e021974d30', vmId='8391f9a7-7fe7-4025-8fba-6273ae587a52'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=1}', managed='false', plugged='true', readOnly='false', deviceAlias='channel0', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}}, vmType=kvm, memSize=1024, smpCoresPerSocket=1, vmName=archive, nice=0, status=Up, maxMemSize=4096, bootMenuEnable=false, pid=6333, smpThreadsPerCore=1, memGuaranteedSize=682, kvmEnable=true, displayNetwork=yorku, devices=[Ljava.lang.Object;@4e7e916, display=qxl, maxVCpus=16, clientIp=, statusTime=5262384430, maxMemSlots=16}], log id: 26f8a931 2017-10-01 16:29:28,165-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler7) [6b3cf907] Received a memballoon Device without an address when processing VM 8391f9a7-7fe7-4025-8fba-6273ae587a52 devices, skipping device: {device=memballoon, specParams={model=none}, type=balloon, alias=balloon0} 2017-10-01 16:29:28,165-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler7) [6b3cf907] Received a spice Device without an address when processing VM 8391f9a7-7fe7-4025-8fba-6273ae587a52 devices, skipping device: {device=spice, specParams={fileTransferEnable=true, displayNetwork=yorku, copyPasteEnable=true, displayIp=130.63.94.36, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,ssmartcard,susbredir}, type=graphics, deviceId=c520628b-2ea8-4b60-b6c6-590abc0e7ef8, tlsPort=5900} 2017-10-01 16:29:56,866-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler7) [309aaca0] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 16:30:30,953-04 INFO [org.ovirt.engine.core.bll.SetVmTicketCommand] (default task-36) [d53d355] Running command: SetVmTicketCommand internal: false. Entities affected : ID: 8391f9a7-7fe7-4025-8fba-6273ae587a52 Type: VMAction group CONNECT_TO_VM with role type USER 2017-10-01 16:30:30,983-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (default task-36) [d53d355] START, SetVmTicketVDSCommand(HostName = virt3, SetVmTicketVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vmId='8391f9a7-7fe7-4025-8fba-6273ae587a52', protocol='SPICE', ticket='LZVyuq6fgVr9', validTime='120', userName='admin', userId='fdfc627c-d875-11e0-90f0-83df133b58cc', disconnectAction='LOCK_SCREEN'}), log id: 66317af5 2017-10-01 16:30:31,561-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (default task-36) [d53d355] FINISH, SetVmTicketVDSCommand, log id: 66317af5 2017-10-01 16:30:31,571-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-36) [d53d355] EVENT_ID: VM_SET_TICKET(164), Correlation ID: d53d355, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: User admin@internal initiated console session for VM archive 2017-10-01 16:30:47,064-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [6900ad29] EVENT_ID: VM_CONSOLE_CONNECTED(167), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: User admin@internal is connected to VM archive. 2017-10-01 16:30:47,065-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler5) [6900ad29] VM '8391f9a7-7fe7-4025-8fba-6273ae587a52'(archive) moved from 'PoweringUp' --> 'Up' 2017-10-01 16:30:47,071-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [6900ad29] EVENT_ID: USER_RUN_VM(32), Correlation ID: b5599ec5-85b5-418d-8d1f-58fa9db827ef, Job ID: d5b73987-8137-4850-adaa-eda2b6308572, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM archive started on Host virt3 2017-10-01 16:30:56,885-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [7232e820] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 16:31:00,479-04 INFO [org.ovirt.engine.core.bll.RunVmCommand] (default task-44) [509ee821-b2e0-4b7a-9157-2901b521595b] Lock Acquired to object 'EngineLock:{exclusiveLocks='[a8d4e149-55ec-4961-a414-d7d1e1b05a59=VM]', sharedLocks=''}' 2017-10-01 16:31:00,497-04 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (default task-44) [509ee821-b2e0-4b7a-9157-2901b521595b] START, IsVmDuringInitiatingVDSCommand( IsVmDuringInitiatingVDSCommandParameters:{runAsync='true', vmId='a8d4e149-55ec-4961-a414-d7d1e1b05a59'}), log id: 3d48c752 2017-10-01 16:31:00,498-04 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (default task-44) [509ee821-b2e0-4b7a-9157-2901b521595b] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 3d48c752 2017-10-01 16:31:00,595-04 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-6-thread-21) [509ee821-b2e0-4b7a-9157-2901b521595b] Running command: RunVmCommand internal: false. Entities affected : ID: a8d4e149-55ec-4961-a414-d7d1e1b05a59 Type: VMAction group RUN_VM with role type USER 2017-10-01 16:31:00,694-04 INFO [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (org.ovirt.thread.pool-6-thread-21) [509ee821-b2e0-4b7a-9157-2901b521595b] START, UpdateVmDynamicDataVDSCommand( UpdateVmDynamicDataVDSCommandParameters:{runAsync='true', hostId='null', vmId='a8d4e149-55ec-4961-a414-d7d1e1b05a59', vmDynamic='org.ovirt.engine.core.common.businessentities.VmDynamic@dae70535'}), log id: 6ebd66c3 2017-10-01 16:31:00,698-04 INFO [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (org.ovirt.thread.pool-6-thread-21) [509ee821-b2e0-4b7a-9157-2901b521595b] FINISH, UpdateVmDynamicDataVDSCommand, log id: 6ebd66c3 2017-10-01 16:31:00,700-04 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-6-thread-21) [509ee821-b2e0-4b7a-9157-2901b521595b] START, CreateVmVDSCommand( CreateVmVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vmId='a8d4e149-55ec-4961-a414-d7d1e1b05a59', vm='VM [fix]'}), log id: 7d43920f 2017-10-01 16:31:00,704-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-6-thread-21) [509ee821-b2e0-4b7a-9157-2901b521595b] START, CreateVDSCommand(HostName = virt1, CreateVmVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vmId='a8d4e149-55ec-4961-a414-d7d1e1b05a59', vm='VM [fix]'}), log id: ba310a5 2017-10-01 16:31:00,707-04 INFO [org.ovirt.engine.core.vdsbroker.builder.vminfo.VmInfoBuilderImpl] (org.ovirt.thread.pool-6-thread-21) [509ee821-b2e0-4b7a-9157-2901b521595b] Bootable disk '462ea313-ff44-462e-b34a-f77ed827b1ec' set to index '0' 2017-10-01 16:31:00,722-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-6-thread-21) [509ee821-b2e0-4b7a-9157-2901b521595b] VM {memGuaranteedSize=682, smpThreadsPerCore=1, cpuType=SandyBridge, vmId=a8d4e149-55ec-4961-a414-d7d1e1b05a59, acpiEnable=true, vmType=kvm, smp=2, smpCoresPerSocket=1, emulatedMachine=pc-i440fx-rhel7.3.0, smartcardEnable=false, guestNumaNodes=[{memory=1024, cpus=0,1, nodeIndex=0}], transparentHugePages=true, displayNetwork=yorku, vmName=fix, maxVCpus=16, kvmEnable=true, devices=[{address={bus=0x00, domain=0x0000, function=0x0, slot=0x02, type=pci}, type=video, specParams={heads=1, vram=32768}, device=qxl, deviceId=65535555-f6ea-4c42-aa22-bde511fd016e}, {type=graphics, specParams={fileTransferEnable=true, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,ssmartcard,susbredir, copyPasteEnable=true}, device=spice, deviceId=35278398-cfdd-45eb-a505-508c32fff597}, {iface=ide, shared=false, path=, address={bus=1, controller=0, unit=0, type=drive, target=0}, readonly=true, index=2, type=disk, specParams={path=}, device=cdrom, deviceId=63716762-926b-41a3-ab82-f2f91a3e7d41}, {discard=false, shared=false, address={bus=0x00, domain=0x0000, function=0x0, slot=0x07, type=pci}, imageID=462ea313-ff44-462e-b34a-f77ed827b1ec, format=raw, index=0, optional=false, type=disk, deviceId=462ea313-ff44-462e-b34a-f77ed827b1ec, domainID=c8653dba-63ec-4447-872c-0ffcc6af9c0f, propagateErrors=off, iface=virtio, readonly=false, bootOrder=1, poolID=aca61eac-ed8a-484b-bbee-3645148f6405, volumeID=3e1aaa3f-45dd-4d83-b11f-969541e535f5, specParams={}, device=disk}, {discard=false, shared=false, address={bus=0x00, domain=0x0000, function=0x0, slot=0x08, type=pci}, imageID=aee67a98-0dfb-434e-b950-908f1b52af73, format=raw, optional=false, type=disk, deviceId=aee67a98-0dfb-434e-b950-908f1b52af73, domainID=c8653dba-63ec-4447-872c-0ffcc6af9c0f, propagateErrors=off, iface=virtio, readonly=false, poolID=aca61eac-ed8a-484b-bbee-3645148f6405, volumeID=f7497416-5fd6-47fd-80cf-dcab7411782c, specParams={}, device=disk}, {filter=vdsm-no-mac-spoofing, nicModel=pv, address={bus=0x00, domain=0x0000, function=0x0, slot=0x03, type=pci}, type=interface, specParams={inbound={}, outbound={}}, device=bridge, linkActive=true, deviceId=b6678d13-51d5-46f1-b382-15796dc20c5e, macAddr=00:1a:4a:32:3d:2e, network=yorku}, {filter=vdsm-no-mac-spoofing, nicModel=pv, address={bus=0x00, domain=0x0000, function=0x0, slot=0x04, type=pci}, type=interface, specParams={inbound={}, outbound={}}, device=bridge, linkActive=true, deviceId=9c072ad4-f132-4849-ad8f-d759c4b4bd1b, macAddr=00:1a:4a:32:3d:2f, network=mrpriv}, {address={bus=0x00, domain=0x0000, function=0x2, slot=0x01, type=pci}, index=0, model=piix3-uhci, type=controller, specParams={}, device=usb, deviceId=e5a77d4e-a45b-4876-8be3-8f6da91a741d}, {address={bus=0x00, domain=0x0000, function=0x0, slot=0x05, type=pci}, index=0, model=virtio-scsi, type=controller, specParams={}, device=scsi, deviceId=e385f7cc-5f4b-470a-ac1a-d6b0965bfb93}, {address={bus=0x00, domain=0x0000, function=0x0, slot=0x06, type=pci}, type=controller, specParams={}, device=virtio-serial, deviceId=a0d5604c-cc52-46ad-a691-ad84f9f3e967}, {model=virtio, type=rng, specParams={source=urandom}, device=virtio, deviceId=f6b5d098-9e58-415d-b5d0-6d478124aae7}], custom={device_1368d112-ae87-4095-8857-5b2327b9eaa2device_c4ccac70-16c7-4645-b14b-8528cd3eac5edevice_62ad7fd2-46b7-4177-87ac-4b45ce3577a0device_af11d3e6-35fb-4d74-8596-5ff651df3079=VmDevice:{id='VmDeviceId:{deviceId='af11d3e6-35fb-4d74-8596-5ff651df3079', vmId='a8d4e149-55ec-4961-a414-d7d1e1b05a59'}', device='ide', type='CONTROLLER', bootOrder='0', specParams='[]', address='{slot=0x01, bus=0x00, domain=0x0000, type=pci, function=0x1}', managed='false', plugged='true', readOnly='false', deviceAlias='ide', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_1368d112-ae87-4095-8857-5b2327b9eaa2device_c4ccac70-16c7-4645-b14b-8528cd3eac5edevice_62ad7fd2-46b7-4177-87ac-4b45ce3577a0=VmDevice:{id='VmDeviceId:{deviceId='62ad7fd2-46b7-4177-87ac-4b45ce3577a0', vmId='a8d4e149-55ec-4961-a414-d7d1e1b05a59'}', device='spicevmc', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=3}', managed='false', plugged='true', readOnly='false', deviceAlias='channel2', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_1368d112-ae87-4095-8857-5b2327b9eaa2device_c4ccac70-16c7-4645-b14b-8528cd3eac5e=VmDevice:{id='VmDeviceId:{deviceId='c4ccac70-16c7-4645-b14b-8528cd3eac5e', vmId='a8d4e149-55ec-4961-a414-d7d1e1b05a59'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=2}', managed='false', plugged='true', readOnly='false', deviceAlias='channel1', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_1368d112-ae87-4095-8857-5b2327b9eaa2=VmDevice:{id='VmDeviceId:{deviceId='1368d112-ae87-4095-8857-5b2327b9eaa2', vmId='a8d4e149-55ec-4961-a414-d7d1e1b05a59'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=1}', managed='false', plugged='true', readOnly='false', deviceAlias='channel0', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}}, display=qxl, timeOffset=0, nice=0, maxMemSize=4096, maxMemSlots=16, bootMenuEnable=false, memSize=1024} 2017-10-01 16:31:00,740-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-6-thread-21) [509ee821-b2e0-4b7a-9157-2901b521595b] FINISH, CreateVDSCommand, log id: ba310a5 2017-10-01 16:31:00,742-04 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-6-thread-21) [509ee821-b2e0-4b7a-9157-2901b521595b] FINISH, CreateVmVDSCommand, return: WaitForLaunch, log id: 7d43920f 2017-10-01 16:31:00,743-04 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-6-thread-21) [509ee821-b2e0-4b7a-9157-2901b521595b] Lock freed to object 'EngineLock:{exclusiveLocks='[a8d4e149-55ec-4961-a414-d7d1e1b05a59=VM]', sharedLocks=''}' 2017-10-01 16:31:00,747-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-21) [509ee821-b2e0-4b7a-9157-2901b521595b] EVENT_ID: USER_STARTED_VM(153), Correlation ID: 509ee821-b2e0-4b7a-9157-2901b521595b, Job ID: abfaf0eb-ad65-45f0-b347-68799bd75bb9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM fix was started by admin@internal (Host: virt1). 2017-10-01 16:31:03,507-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmsStatisticsFetcher] (DefaultQuartzScheduler1) [312cde0c] Fetched 1 VMs from VDS '45f8b331-842e-48e7-9df8-56adddb93836' 2017-10-01 16:31:03,510-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVDSCommand] (DefaultQuartzScheduler1) [312cde0c] START, FullListVDSCommand(HostName = virt1, FullListVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vmIds='[a8d4e149-55ec-4961-a414-d7d1e1b05a59]'}), log id: 5a0181bb 2017-10-01 16:31:03,984-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVDSCommand] (DefaultQuartzScheduler1) [312cde0c] FINISH, FullListVDSCommand, return: [{acpiEnable=true, emulatedMachine=pc-i440fx-rhel7.3.0, vmId=a8d4e149-55ec-4961-a414-d7d1e1b05a59, guestDiskMapping={}, transparentHugePages=true, timeOffset=0, cpuType=SandyBridge, smp=2, pauseCode=NOERR, guestNumaNodes=[Ljava.lang.Object;@67d32c36, smartcardEnable=false, custom={device_1368d112-ae87-4095-8857-5b2327b9eaa2device_c4ccac70-16c7-4645-b14b-8528cd3eac5edevice_62ad7fd2-46b7-4177-87ac-4b45ce3577a0device_af11d3e6-35fb-4d74-8596-5ff651df3079=VmDevice:{id='VmDeviceId:{deviceId='af11d3e6-35fb-4d74-8596-5ff651df3079', vmId='a8d4e149-55ec-4961-a414-d7d1e1b05a59'}', device='ide', type='CONTROLLER', bootOrder='0', specParams='[]', address='{slot=0x01, bus=0x00, domain=0x0000, type=pci, function=0x1}', managed='false', plugged='true', readOnly='false', deviceAlias='ide', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_1368d112-ae87-4095-8857-5b2327b9eaa2device_c4ccac70-16c7-4645-b14b-8528cd3eac5e=VmDevice:{id='VmDeviceId:{deviceId='c4ccac70-16c7-4645-b14b-8528cd3eac5e', vmId='a8d4e149-55ec-4961-a414-d7d1e1b05a59'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=2}', managed='false', plugged='true', readOnly='false', deviceAlias='channel1', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_1368d112-ae87-4095-8857-5b2327b9eaa2=VmDevice:{id='VmDeviceId:{deviceId='1368d112-ae87-4095-8857-5b2327b9eaa2', vmId='a8d4e149-55ec-4961-a414-d7d1e1b05a59'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=1}', managed='false', plugged='true', readOnly='false', deviceAlias='channel0', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_1368d112-ae87-4095-8857-5b2327b9eaa2device_c4ccac70-16c7-4645-b14b-8528cd3eac5edevice_62ad7fd2-46b7-4177-87ac-4b45ce3577a0=VmDevice:{id='VmDeviceId:{deviceId='62ad7fd2-46b7-4177-87ac-4b45ce3577a0', vmId='a8d4e149-55ec-4961-a414-d7d1e1b05a59'}', device='spicevmc', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=3}', managed='false', plugged='true', readOnly='false', deviceAlias='channel2', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}}, vmType=kvm, memSize=1024, smpCoresPerSocket=1, vmName=fix, nice=0, status=Up, maxMemSize=4096, bootMenuEnable=false, pid=4161, smpThreadsPerCore=1, memGuaranteedSize=682, kvmEnable=true, displayNetwork=yorku, devices=[Ljava.lang.Object;@14037501, display=qxl, maxVCpus=16, clientIp=, statusTime=4295296710, maxMemSlots=16}], log id: 5a0181bb 2017-10-01 16:31:03,987-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler1) [312cde0c] Received a memballoon Device without an address when processing VM a8d4e149-55ec-4961-a414-d7d1e1b05a59 devices, skipping device: {device=memballoon, specParams={model=none}, type=balloon, alias=balloon0} 2017-10-01 16:31:03,987-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler1) [312cde0c] Received a spice Device without an address when processing VM a8d4e149-55ec-4961-a414-d7d1e1b05a59 devices, skipping device: {device=spice, specParams={fileTransferEnable=true, displayNetwork=yorku, copyPasteEnable=true, displayIp=130.63.94.34, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,ssmartcard,susbredir}, type=graphics, deviceId=35278398-cfdd-45eb-a505-508c32fff597, tlsPort=5900} 2017-10-01 16:31:14,933-04 INFO [org.ovirt.engine.core.bll.RunVmCommand] (default task-77) [339336f6-ec1b-4d8c-b747-75194de8f838] Lock Acquired to object 'EngineLock:{exclusiveLocks='[899e3a1c-dcc4-4426-9dd4-0f1e8b94a2b8=VM]', sharedLocks=''}' 2017-10-01 16:31:14,950-04 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (default task-77) [339336f6-ec1b-4d8c-b747-75194de8f838] START, IsVmDuringInitiatingVDSCommand( IsVmDuringInitiatingVDSCommandParameters:{runAsync='true', vmId='899e3a1c-dcc4-4426-9dd4-0f1e8b94a2b8'}), log id: 152e1dcc 2017-10-01 16:31:14,950-04 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (default task-77) [339336f6-ec1b-4d8c-b747-75194de8f838] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 152e1dcc 2017-10-01 16:31:15,046-04 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-6-thread-24) [339336f6-ec1b-4d8c-b747-75194de8f838] Running command: RunVmCommand internal: false. Entities affected : ID: 899e3a1c-dcc4-4426-9dd4-0f1e8b94a2b8 Type: VMAction group RUN_VM with role type USER 2017-10-01 16:31:15,130-04 INFO [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (org.ovirt.thread.pool-6-thread-24) [339336f6-ec1b-4d8c-b747-75194de8f838] START, UpdateVmDynamicDataVDSCommand( UpdateVmDynamicDataVDSCommandParameters:{runAsync='true', hostId='null', vmId='899e3a1c-dcc4-4426-9dd4-0f1e8b94a2b8', vmDynamic='org.ovirt.engine.core.common.businessentities.VmDynamic@2f332295'}), log id: 76f84fff 2017-10-01 16:31:15,132-04 INFO [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (org.ovirt.thread.pool-6-thread-24) [339336f6-ec1b-4d8c-b747-75194de8f838] FINISH, UpdateVmDynamicDataVDSCommand, log id: 76f84fff 2017-10-01 16:31:15,134-04 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-6-thread-24) [339336f6-ec1b-4d8c-b747-75194de8f838] START, CreateVmVDSCommand( CreateVmVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vmId='899e3a1c-dcc4-4426-9dd4-0f1e8b94a2b8', vm='VM [ipa]'}), log id: 158df9dc 2017-10-01 16:31:15,136-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-6-thread-24) [339336f6-ec1b-4d8c-b747-75194de8f838] START, CreateVDSCommand(HostName = virt2, CreateVmVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vmId='899e3a1c-dcc4-4426-9dd4-0f1e8b94a2b8', vm='VM [ipa]'}), log id: 327dac49 2017-10-01 16:31:15,140-04 INFO [org.ovirt.engine.core.vdsbroker.builder.vminfo.VmInfoBuilderImpl] (org.ovirt.thread.pool-6-thread-24) [339336f6-ec1b-4d8c-b747-75194de8f838] Bootable disk '9539d186-7e3a-4e78-8e6a-c8428e12038a' set to index '0' 2017-10-01 16:31:15,154-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-6-thread-24) [339336f6-ec1b-4d8c-b747-75194de8f838] VM {memGuaranteedSize=2730, smpThreadsPerCore=1, cpuType=SandyBridge, vmId=899e3a1c-dcc4-4426-9dd4-0f1e8b94a2b8, acpiEnable=true, vmType=kvm, smp=2, smpCoresPerSocket=1, emulatedMachine=pc-i440fx-rhel7.3.0, smartcardEnable=false, guestNumaNodes=[{memory=4096, cpus=0,1, nodeIndex=0}], transparentHugePages=true, displayNetwork=yorku, vmName=ipa, maxVCpus=16, kvmEnable=true, devices=[{address={bus=0x00, domain=0x0000, function=0x0, slot=0x02, type=pci}, type=video, specParams={heads=1, vram=32768}, device=qxl, deviceId=96b2b9c1-ff07-437c-ace0-2c48f26c1c4a}, {type=graphics, specParams={fileTransferEnable=true, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,ssmartcard,susbredir, copyPasteEnable=true}, device=spice, deviceId=00000023-0023-0023-0023-00000000016f}, {iface=ide, shared=false, path=, address={bus=1, controller=0, unit=0, type=drive, target=0}, readonly=true, index=2, type=disk, specParams={path=}, device=cdrom, deviceId=a8396db4-c079-4f2d-be11-08365852fff7}, {discard=false, shared=false, address={bus=0x00, domain=0x0000, function=0x0, slot=0x07, type=pci}, imageID=9539d186-7e3a-4e78-8e6a-c8428e12038a, format=raw, index=0, optional=false, type=disk, deviceId=9539d186-7e3a-4e78-8e6a-c8428e12038a, domainID=c8653dba-63ec-4447-872c-0ffcc6af9c0f, propagateErrors=off, iface=virtio, readonly=false, bootOrder=1, poolID=aca61eac-ed8a-484b-bbee-3645148f6405, volumeID=11d3a2bb-ae61-4194-9773-85a927a02c99, specParams={}, device=disk}, {discard=false, shared=false, address={bus=0x00, domain=0x0000, function=0x0, slot=0x08, type=pci}, imageID=07ee6326-1f1c-4105-b544-569e2f31a19c, format=raw, optional=false, type=disk, deviceId=07ee6326-1f1c-4105-b544-569e2f31a19c, domainID=c8653dba-63ec-4447-872c-0ffcc6af9c0f, propagateErrors=off, iface=virtio, readonly=false, poolID=aca61eac-ed8a-484b-bbee-3645148f6405, volumeID=6436ea04-c3e6-4559-b514-a195c31ac3b2, specParams={}, device=disk}, {filter=vdsm-no-mac-spoofing, nicModel=pv, address={bus=0x00, domain=0x0000, function=0x0, slot=0x03, type=pci}, type=interface, specParams={inbound={}, outbound={}}, device=bridge, linkActive=true, deviceId=09d8977f-cfb4-43aa-8dfd-c25b0ca28e9e, macAddr=00:1a:4a:32:3d:04, network=yorku}, {filter=vdsm-no-mac-spoofing, nicModel=pv, address={bus=0x00, domain=0x0000, function=0x0, slot=0x04, type=pci}, type=interface, specParams={inbound={}, outbound={}}, device=bridge, linkActive=true, deviceId=bb1589ac-67f3-4566-8565-50c326e223af, macAddr=00:1a:4a:32:3d:05, network=mrpriv}, {address={bus=0x00, domain=0x0000, function=0x2, slot=0x01, type=pci}, index=0, model=piix3-uhci, type=controller, specParams={}, device=usb, deviceId=2c5cabe2-6e6a-4bf0-80d0-650143c8fbd4}, {address={bus=0x00, function=0x0, domain=0x0000, slot=0x09, type=pci}, type=balloon, specParams={model=virtio}, device=memballoon, deviceId=020cd4e7-e968-4a50-b5f6-f8f97d1b8f1e}, {address={bus=0x00, domain=0x0000, function=0x0, slot=0x05, type=pci}, index=0, model=virtio-scsi, type=controller, specParams={}, device=scsi, deviceId=da355dd2-e202-4607-b809-b1146308f178}, {address={bus=0x00, domain=0x0000, function=0x0, slot=0x06, type=pci}, type=controller, specParams={}, device=virtio-serial, deviceId=0a99fd0d-d2e0-4662-85d6-2b6a7bf49c27}, {model=virtio, type=rng, specParams={source=urandom}, device=virtio, deviceId=c1c1784f-d84a-4c64-b472-d7b5364d3e91}], custom={device_c4ca6d01-88f1-4cd5-ba37-ce543be95f2fdevice_bf4bfb99-4556-4b45-af2b-38af29ab24e5device_8a92516a-5882-4ea5-96c2-1e1f7d6c95d5=VmDevice:{id='VmDeviceId:{deviceId='8a92516a-5882-4ea5-96c2-1e1f7d6c95d5', vmId='899e3a1c-dcc4-4426-9dd4-0f1e8b94a2b8'}', device='spicevmc', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=3}', managed='false', plugged='true', readOnly='false', deviceAlias='channel2', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_c4ca6d01-88f1-4cd5-ba37-ce543be95f2fdevice_bf4bfb99-4556-4b45-af2b-38af29ab24e5=VmDevice:{id='VmDeviceId:{deviceId='bf4bfb99-4556-4b45-af2b-38af29ab24e5', vmId='899e3a1c-dcc4-4426-9dd4-0f1e8b94a2b8'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=2}', managed='false', plugged='true', readOnly='false', deviceAlias='channel1', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_c4ca6d01-88f1-4cd5-ba37-ce543be95f2f=VmDevice:{id='VmDeviceId:{deviceId='c4ca6d01-88f1-4cd5-ba37-ce543be95f2f', vmId='899e3a1c-dcc4-4426-9dd4-0f1e8b94a2b8'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=1}', managed='false', plugged='true', readOnly='false', deviceAlias='channel0', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_c4ca6d01-88f1-4cd5-ba37-ce543be95f2fdevice_bf4bfb99-4556-4b45-af2b-38af29ab24e5device_8a92516a-5882-4ea5-96c2-1e1f7d6c95d5device_69c8a952-095f-4458-b7e9-794af0d7ad68=VmDevice:{id='VmDeviceId:{deviceId='69c8a952-095f-4458-b7e9-794af0d7ad68', vmId='899e3a1c-dcc4-4426-9dd4-0f1e8b94a2b8'}', device='ide', type='CONTROLLER', bootOrder='0', specParams='[]', address='{slot=0x01, bus=0x00, domain=0x0000, type=pci, function=0x1}', managed='false', plugged='true', readOnly='false', deviceAlias='ide', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}}, display=qxl, timeOffset=0, nice=0, maxMemSize=16384, maxMemSlots=16, bootMenuEnable=false, memSize=4096} 2017-10-01 16:31:15,835-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-6-thread-24) [339336f6-ec1b-4d8c-b747-75194de8f838] FINISH, CreateVDSCommand, log id: 327dac49 2017-10-01 16:31:15,838-04 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-6-thread-24) [339336f6-ec1b-4d8c-b747-75194de8f838] FINISH, CreateVmVDSCommand, return: WaitForLaunch, log id: 158df9dc 2017-10-01 16:31:15,839-04 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-6-thread-24) [339336f6-ec1b-4d8c-b747-75194de8f838] Lock freed to object 'EngineLock:{exclusiveLocks='[899e3a1c-dcc4-4426-9dd4-0f1e8b94a2b8=VM]', sharedLocks=''}' 2017-10-01 16:31:15,844-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-24) [339336f6-ec1b-4d8c-b747-75194de8f838] EVENT_ID: USER_STARTED_VM(153), Correlation ID: 339336f6-ec1b-4d8c-b747-75194de8f838, Job ID: 29634791-24f1-4105-a1fd-96faa53658da, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM ipa was started by admin@internal (Host: virt2). 2017-10-01 16:31:19,318-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler8) [25d81b69] VM 'a8d4e149-55ec-4961-a414-d7d1e1b05a59'(fix) moved from 'WaitForLaunch' --> 'PoweringUp' 2017-10-01 16:31:19,324-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVDSCommand] (DefaultQuartzScheduler8) [25d81b69] START, FullListVDSCommand(HostName = virt1, FullListVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vmIds='[a8d4e149-55ec-4961-a414-d7d1e1b05a59]'}), log id: 7ebee9af 2017-10-01 16:31:19,347-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVDSCommand] (DefaultQuartzScheduler8) [25d81b69] FINISH, FullListVDSCommand, return: [{acpiEnable=true, emulatedMachine=pc-i440fx-rhel7.3.0, vmId=a8d4e149-55ec-4961-a414-d7d1e1b05a59, guestDiskMapping={}, transparentHugePages=true, timeOffset=0, cpuType=SandyBridge, smp=2, pauseCode=NOERR, guestNumaNodes=[Ljava.lang.Object;@5f3888c9, smartcardEnable=false, custom={device_1368d112-ae87-4095-8857-5b2327b9eaa2device_c4ccac70-16c7-4645-b14b-8528cd3eac5edevice_62ad7fd2-46b7-4177-87ac-4b45ce3577a0device_af11d3e6-35fb-4d74-8596-5ff651df3079=VmDevice:{id='VmDeviceId:{deviceId='af11d3e6-35fb-4d74-8596-5ff651df3079', vmId='a8d4e149-55ec-4961-a414-d7d1e1b05a59'}', device='ide', type='CONTROLLER', bootOrder='0', specParams='[]', address='{slot=0x01, bus=0x00, domain=0x0000, type=pci, function=0x1}', managed='false', plugged='true', readOnly='false', deviceAlias='ide', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_1368d112-ae87-4095-8857-5b2327b9eaa2device_c4ccac70-16c7-4645-b14b-8528cd3eac5e=VmDevice:{id='VmDeviceId:{deviceId='c4ccac70-16c7-4645-b14b-8528cd3eac5e', vmId='a8d4e149-55ec-4961-a414-d7d1e1b05a59'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=2}', managed='false', plugged='true', readOnly='false', deviceAlias='channel1', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_1368d112-ae87-4095-8857-5b2327b9eaa2=VmDevice:{id='VmDeviceId:{deviceId='1368d112-ae87-4095-8857-5b2327b9eaa2', vmId='a8d4e149-55ec-4961-a414-d7d1e1b05a59'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=1}', managed='false', plugged='true', readOnly='false', deviceAlias='channel0', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_1368d112-ae87-4095-8857-5b2327b9eaa2device_c4ccac70-16c7-4645-b14b-8528cd3eac5edevice_62ad7fd2-46b7-4177-87ac-4b45ce3577a0=VmDevice:{id='VmDeviceId:{deviceId='62ad7fd2-46b7-4177-87ac-4b45ce3577a0', vmId='a8d4e149-55ec-4961-a414-d7d1e1b05a59'}', device='spicevmc', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=3}', managed='false', plugged='true', readOnly='false', deviceAlias='channel2', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}}, vmType=kvm, memSize=1024, smpCoresPerSocket=1, vmName=fix, nice=0, status=Up, maxMemSize=4096, bootMenuEnable=false, pid=4161, smpThreadsPerCore=1, memGuaranteedSize=682, kvmEnable=true, displayNetwork=yorku, devices=[Ljava.lang.Object;@6299ff2a, display=qxl, maxVCpus=16, clientIp=, statusTime=4295312520, maxMemSlots=16}], log id: 7ebee9af 2017-10-01 16:31:19,347-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler8) [25d81b69] Received a memballoon Device without an address when processing VM a8d4e149-55ec-4961-a414-d7d1e1b05a59 devices, skipping device: {device=memballoon, specParams={model=none}, type=balloon, alias=balloon0} 2017-10-01 16:31:19,348-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler8) [25d81b69] Received a spice Device without an address when processing VM a8d4e149-55ec-4961-a414-d7d1e1b05a59 devices, skipping device: {device=spice, specParams={fileTransferEnable=true, displayNetwork=yorku, copyPasteEnable=true, displayIp=130.63.94.34, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,ssmartcard,susbredir}, type=graphics, deviceId=35278398-cfdd-45eb-a505-508c32fff597, tlsPort=5900} 2017-10-01 16:31:19,563-04 INFO [org.ovirt.engine.core.bll.RunVmCommand] (default task-88) [a581bf39-0479-4cab-b2b7-0a39de2024cf] Lock Acquired to object 'EngineLock:{exclusiveLocks='[db3e08fb-c111-442d-aeec-1507da30f486=VM]', sharedLocks=''}' 2017-10-01 16:31:19,580-04 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (default task-88) [a581bf39-0479-4cab-b2b7-0a39de2024cf] START, IsVmDuringInitiatingVDSCommand( IsVmDuringInitiatingVDSCommandParameters:{runAsync='true', vmId='db3e08fb-c111-442d-aeec-1507da30f486'}), log id: 14abb40 2017-10-01 16:31:19,580-04 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (default task-88) [a581bf39-0479-4cab-b2b7-0a39de2024cf] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 14abb40 2017-10-01 16:31:19,659-04 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-6-thread-32) [a581bf39-0479-4cab-b2b7-0a39de2024cf] Running command: RunVmCommand internal: false. Entities affected : ID: db3e08fb-c111-442d-aeec-1507da30f486 Type: VMAction group RUN_VM with role type USER 2017-10-01 16:31:19,753-04 INFO [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (org.ovirt.thread.pool-6-thread-32) [a581bf39-0479-4cab-b2b7-0a39de2024cf] START, UpdateVmDynamicDataVDSCommand( UpdateVmDynamicDataVDSCommandParameters:{runAsync='true', hostId='null', vmId='db3e08fb-c111-442d-aeec-1507da30f486', vmDynamic='org.ovirt.engine.core.common.businessentities.VmDynamic@3b1fd825'}), log id: 4292813b 2017-10-01 16:31:19,756-04 INFO [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (org.ovirt.thread.pool-6-thread-32) [a581bf39-0479-4cab-b2b7-0a39de2024cf] FINISH, UpdateVmDynamicDataVDSCommand, log id: 4292813b 2017-10-01 16:31:19,758-04 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-6-thread-32) [a581bf39-0479-4cab-b2b7-0a39de2024cf] START, CreateVmVDSCommand( CreateVmVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vmId='db3e08fb-c111-442d-aeec-1507da30f486', vm='VM [print]'}), log id: 7b1aa7aa 2017-10-01 16:31:19,760-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-6-thread-32) [a581bf39-0479-4cab-b2b7-0a39de2024cf] START, CreateVDSCommand(HostName = virt4, CreateVmVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vmId='db3e08fb-c111-442d-aeec-1507da30f486', vm='VM [print]'}), log id: 639526d7 2017-10-01 16:31:19,763-04 INFO [org.ovirt.engine.core.vdsbroker.builder.vminfo.VmInfoBuilderImpl] (org.ovirt.thread.pool-6-thread-32) [a581bf39-0479-4cab-b2b7-0a39de2024cf] Bootable disk '2aace45d-dcc5-4b74-b07d-1c24d515be33' set to index '0' 2017-10-01 16:31:19,775-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-6-thread-32) [a581bf39-0479-4cab-b2b7-0a39de2024cf] VM {memGuaranteedSize=4096, smpThreadsPerCore=1, cpuType=SandyBridge, vmId=db3e08fb-c111-442d-aeec-1507da30f486, acpiEnable=true, vmType=kvm, smp=2, smpCoresPerSocket=1, emulatedMachine=pc-i440fx-rhel7.3.0, smartcardEnable=false, guestNumaNodes=[{memory=4096, cpus=0,1, nodeIndex=0}], transparentHugePages=true, displayNetwork=yorku, vmName=print, maxVCpus=16, kvmEnable=true, devices=[{address={bus=0x00, domain=0x0000, function=0x0, slot=0x02, type=pci}, type=video, specParams={heads=1, vram=32768}, device=qxl, deviceId=abd14dab-c82b-4213-8add-a25beb5fc9fb}, {type=graphics, specParams={fileTransferEnable=true, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,ssmartcard,susbredir, copyPasteEnable=true}, device=spice, deviceId=00000035-0035-0035-0035-0000000003a5}, {iface=ide, shared=false, path=, address={bus=1, controller=0, unit=0, type=drive, target=0}, readonly=true, index=2, type=disk, specParams={path=}, device=cdrom, deviceId=dd57c683-f138-452d-862c-a13e0da43228}, {discard=false, shared=false, address={bus=0x00, domain=0x0000, function=0x0, slot=0x07, type=pci}, imageID=2aace45d-dcc5-4b74-b07d-1c24d515be33, format=raw, index=0, optional=false, type=disk, deviceId=2aace45d-dcc5-4b74-b07d-1c24d515be33, domainID=c8653dba-63ec-4447-872c-0ffcc6af9c0f, propagateErrors=off, iface=virtio, readonly=false, bootOrder=1, poolID=aca61eac-ed8a-484b-bbee-3645148f6405, volumeID=8f18f657-8645-492e-98a9-d0e63e4b8b1c, specParams={}, device=disk}, {filter=vdsm-no-mac-spoofing, nicModel=pv, address={bus=0x00, domain=0x0000, function=0x0, slot=0x03, type=pci}, type=interface, specParams={inbound={}, outbound={}}, device=bridge, linkActive=true, deviceId=99d6660b-9a42-4efb-aae6-a703576cb915, macAddr=00:1a:4a:32:3d:1d, network=yorku}, {filter=vdsm-no-mac-spoofing, nicModel=pv, address={bus=0x00, domain=0x0000, function=0x0, slot=0x04, type=pci}, type=interface, specParams={inbound={}, outbound={}}, device=bridge, linkActive=true, deviceId=244dcebf-4629-4383-b7f6-12a18f7cc3d4, macAddr=00:1a:4a:32:3d:1e, network=mrpriv}, {address={bus=0x00, domain=0x0000, function=0x2, slot=0x01, type=pci}, index=0, model=piix3-uhci, type=controller, specParams={}, device=usb, deviceId=6e450231-6e44-414d-b3a5-03137ccf903f}, {address={bus=0x00, domain=0x0000, function=0x0, slot=0x05, type=pci}, index=0, model=virtio-scsi, type=controller, specParams={}, device=scsi, deviceId=4c545587-12ad-42cb-bebc-4be91558dc0f}, {address={bus=0x00, domain=0x0000, function=0x0, slot=0x06, type=pci}, type=controller, specParams={}, device=virtio-serial, deviceId=26e5aa2e-7ddb-4c8e-929d-0deeaddebe25}, {model=virtio, type=rng, specParams={source=urandom}, device=virtio, deviceId=4ad59492-4e16-4e97-a9ef-ab7e037c6067}], custom={device_547649a9-4bf9-49b8-91a4-12dfb46dab53device_b8489d3b-06ad-4649-b6af-26f137b2b702=VmDevice:{id='VmDeviceId:{deviceId='b8489d3b-06ad-4649-b6af-26f137b2b702', vmId='db3e08fb-c111-442d-aeec-1507da30f486'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=2}', managed='false', plugged='true', readOnly='false', deviceAlias='channel1', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_547649a9-4bf9-49b8-91a4-12dfb46dab53=VmDevice:{id='VmDeviceId:{deviceId='547649a9-4bf9-49b8-91a4-12dfb46dab53', vmId='db3e08fb-c111-442d-aeec-1507da30f486'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=1}', managed='false', plugged='true', readOnly='false', deviceAlias='channel0', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_547649a9-4bf9-49b8-91a4-12dfb46dab53device_b8489d3b-06ad-4649-b6af-26f137b2b702device_7bf8fe87-fe28-47a4-907c-b0665ad11d14device_296d9682-1520-452e-974e-15e1b8a69dd6=VmDevice:{id='VmDeviceId:{deviceId='296d9682-1520-452e-974e-15e1b8a69dd6', vmId='db3e08fb-c111-442d-aeec-1507da30f486'}', device='ide', type='CONTROLLER', bootOrder='0', specParams='[]', address='{slot=0x01, bus=0x00, domain=0x0000, type=pci, function=0x1}', managed='false', plugged='true', readOnly='false', deviceAlias='ide', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_547649a9-4bf9-49b8-91a4-12dfb46dab53device_b8489d3b-06ad-4649-b6af-26f137b2b702device_7bf8fe87-fe28-47a4-907c-b0665ad11d14=VmDevice:{id='VmDeviceId:{deviceId='7bf8fe87-fe28-47a4-907c-b0665ad11d14', vmId='db3e08fb-c111-442d-aeec-1507da30f486'}', device='spicevmc', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=3}', managed='false', plugged='true', readOnly='false', deviceAlias='channel2', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}}, display=qxl, timeOffset=0, nice=0, maxMemSize=16384, maxMemSlots=16, bootMenuEnable=false, memSize=4096} 2017-10-01 16:31:19,787-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-6-thread-32) [a581bf39-0479-4cab-b2b7-0a39de2024cf] FINISH, CreateVDSCommand, log id: 639526d7 2017-10-01 16:31:19,798-04 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-6-thread-32) [a581bf39-0479-4cab-b2b7-0a39de2024cf] FINISH, CreateVmVDSCommand, return: WaitForLaunch, log id: 7b1aa7aa 2017-10-01 16:31:19,799-04 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-6-thread-32) [a581bf39-0479-4cab-b2b7-0a39de2024cf] Lock freed to object 'EngineLock:{exclusiveLocks='[db3e08fb-c111-442d-aeec-1507da30f486=VM]', sharedLocks=''}' 2017-10-01 16:31:19,806-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-32) [a581bf39-0479-4cab-b2b7-0a39de2024cf] EVENT_ID: USER_STARTED_VM(153), Correlation ID: a581bf39-0479-4cab-b2b7-0a39de2024cf, Job ID: c308d1c2-d2d6-4237-80c0-0532d52c100a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM print was started by admin@internal (Host: virt4). 2017-10-01 16:31:25,902-04 INFO [org.ovirt.engine.core.bll.RunVmCommand] (default task-116) [7c32dd71-7364-4968-b061-bfc1546ad1ed] Lock Acquired to object 'EngineLock:{exclusiveLocks='[5885c43a-b133-45dc-b8d5-1aa2a4a19579=VM]', sharedLocks=''}' 2017-10-01 16:31:25,919-04 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (default task-116) [7c32dd71-7364-4968-b061-bfc1546ad1ed] START, IsVmDuringInitiatingVDSCommand( IsVmDuringInitiatingVDSCommandParameters:{runAsync='true', vmId='5885c43a-b133-45dc-b8d5-1aa2a4a19579'}), log id: 40ea43a5 2017-10-01 16:31:25,919-04 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (default task-116) [7c32dd71-7364-4968-b061-bfc1546ad1ed] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 40ea43a5 2017-10-01 16:31:26,002-04 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-6-thread-44) [7c32dd71-7364-4968-b061-bfc1546ad1ed] Running command: RunVmCommand internal: false. Entities affected : ID: 5885c43a-b133-45dc-b8d5-1aa2a4a19579 Type: VMAction group RUN_VM with role type USER 2017-10-01 16:31:26,086-04 INFO [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (org.ovirt.thread.pool-6-thread-44) [7c32dd71-7364-4968-b061-bfc1546ad1ed] START, UpdateVmDynamicDataVDSCommand( UpdateVmDynamicDataVDSCommandParameters:{runAsync='true', hostId='null', vmId='5885c43a-b133-45dc-b8d5-1aa2a4a19579', vmDynamic='org.ovirt.engine.core.common.businessentities.VmDynamic@7045f095'}), log id: 4a921c5d 2017-10-01 16:31:26,088-04 INFO [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (org.ovirt.thread.pool-6-thread-44) [7c32dd71-7364-4968-b061-bfc1546ad1ed] FINISH, UpdateVmDynamicDataVDSCommand, log id: 4a921c5d 2017-10-01 16:31:26,089-04 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-6-thread-44) [7c32dd71-7364-4968-b061-bfc1546ad1ed] START, CreateVmVDSCommand( CreateVmVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vmId='5885c43a-b133-45dc-b8d5-1aa2a4a19579', vm='VM [prs]'}), log id: 2077842b 2017-10-01 16:31:26,091-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-6-thread-44) [7c32dd71-7364-4968-b061-bfc1546ad1ed] START, CreateVDSCommand(HostName = virt3, CreateVmVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vmId='5885c43a-b133-45dc-b8d5-1aa2a4a19579', vm='VM [prs]'}), log id: 4f99e3b 2017-10-01 16:31:26,094-04 INFO [org.ovirt.engine.core.vdsbroker.builder.vminfo.VmInfoBuilderImpl] (org.ovirt.thread.pool-6-thread-44) [7c32dd71-7364-4968-b061-bfc1546ad1ed] Bootable disk '582e076e-e836-49b3-bf82-28761eb19ce8' set to index '0' 2017-10-01 16:31:26,105-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-6-thread-44) [7c32dd71-7364-4968-b061-bfc1546ad1ed] VM {memGuaranteedSize=682, smpThreadsPerCore=1, cpuType=SandyBridge, vmId=5885c43a-b133-45dc-b8d5-1aa2a4a19579, acpiEnable=true, vmType=kvm, smp=1, smpCoresPerSocket=1, emulatedMachine=pc-i440fx-rhel7.3.0, smartcardEnable=false, guestNumaNodes=[{memory=1024, cpus=0, nodeIndex=0}], transparentHugePages=true, displayNetwork=yorku, vmName=prs, maxVCpus=16, kvmEnable=true, devices=[{address={bus=0x00, domain=0x0000, function=0x0, slot=0x02, type=pci}, type=video, specParams={heads=1, vram=32768}, device=qxl, deviceId=83864d99-1316-414b-8328-1ac58c4b252d}, {type=graphics, specParams={fileTransferEnable=true, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,ssmartcard,susbredir, copyPasteEnable=true}, device=spice, deviceId=8c3d4408-d4ef-44d2-9442-b048e1668bee}, {iface=ide, shared=false, path=, address={bus=1, controller=0, unit=0, type=drive, target=0}, readonly=true, index=2, type=disk, specParams={path=}, device=cdrom, deviceId=07b8197b-a784-4e0f-bb3b-8307a78da2c3}, {discard=false, shared=false, address={bus=0x00, domain=0x0000, function=0x0, slot=0x06, type=pci}, imageID=582e076e-e836-49b3-bf82-28761eb19ce8, format=cow, index=0, optional=false, type=disk, deviceId=582e076e-e836-49b3-bf82-28761eb19ce8, domainID=c8653dba-63ec-4447-872c-0ffcc6af9c0f, propagateErrors=off, iface=virtio, readonly=false, bootOrder=1, poolID=aca61eac-ed8a-484b-bbee-3645148f6405, volumeID=afe4c430-318f-4c8c-9a2c-330a25754d22, specParams={}, device=disk}, {filter=vdsm-no-mac-spoofing, nicModel=pv, address={bus=0x00, domain=0x0000, function=0x0, slot=0x03, type=pci}, type=interface, specParams={inbound={}, outbound={}}, device=bridge, linkActive=true, deviceId=5cb3d977-84f8-4f41-a0a2-272b747ad872, macAddr=00:1a:4a:32:3d:2b, network=yorku}, {address={bus=0x00, domain=0x0000, function=0x2, slot=0x01, type=pci}, index=0, model=piix3-uhci, type=controller, specParams={}, device=usb, deviceId=a7134198-4151-4f9e-9909-9a57178b458a}, {address={bus=0x00, domain=0x0000, function=0x0, slot=0x04, type=pci}, index=0, model=virtio-scsi, type=controller, specParams={}, device=scsi, deviceId=c43e6be1-8555-4fad-8892-4463f2301ac9}, {address={bus=0x00, domain=0x0000, function=0x0, slot=0x05, type=pci}, type=controller, specParams={}, device=virtio-serial, deviceId=90cfa5d4-d06b-4c65-aa84-3a45b139f4b2}, {model=virtio, type=rng, specParams={source=urandom}, device=virtio, deviceId=f2565e19-452b-4e3d-8504-ed48fd66bca2}], custom={device_83673bc9-c843-4588-854d-5166d505dbbadevice_9494c9a1-5388-4595-bdac-a2cdfe926c8ddevice_d8c69979-7261-4dd5-b667-01d2e92187f4=VmDevice:{id='VmDeviceId:{deviceId='d8c69979-7261-4dd5-b667-01d2e92187f4', vmId='5885c43a-b133-45dc-b8d5-1aa2a4a19579'}', device='spicevmc', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=3}', managed='false', plugged='true', readOnly='false', deviceAlias='channel2', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_83673bc9-c843-4588-854d-5166d505dbbadevice_9494c9a1-5388-4595-bdac-a2cdfe926c8d=VmDevice:{id='VmDeviceId:{deviceId='9494c9a1-5388-4595-bdac-a2cdfe926c8d', vmId='5885c43a-b133-45dc-b8d5-1aa2a4a19579'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=2}', managed='false', plugged='true', readOnly='false', deviceAlias='channel1', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_83673bc9-c843-4588-854d-5166d505dbba=VmDevice:{id='VmDeviceId:{deviceId='83673bc9-c843-4588-854d-5166d505dbba', vmId='5885c43a-b133-45dc-b8d5-1aa2a4a19579'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=1}', managed='false', plugged='true', readOnly='false', deviceAlias='channel0', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_83673bc9-c843-4588-854d-5166d505dbbadevice_9494c9a1-5388-4595-bdac-a2cdfe926c8ddevice_d8c69979-7261-4dd5-b667-01d2e92187f4device_1f01430f-3702-4ec6-83e1-fdb9a730f67e=VmDevice:{id='VmDeviceId:{deviceId='1f01430f-3702-4ec6-83e1-fdb9a730f67e', vmId='5885c43a-b133-45dc-b8d5-1aa2a4a19579'}', device='ide', type='CONTROLLER', bootOrder='0', specParams='[]', address='{slot=0x01, bus=0x00, domain=0x0000, type=pci, function=0x1}', managed='false', plugged='true', readOnly='false', deviceAlias='ide', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}}, display=qxl, timeOffset=0, nice=0, maxMemSize=4096, maxMemSlots=16, bootMenuEnable=false, memSize=1024} 2017-10-01 16:31:26,106-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmsStatisticsFetcher] (DefaultQuartzScheduler7) [3cc45c32] Fetched 1 VMs from VDS 'a0ccfd11-3402-4eba-a119-4987561b738e' 2017-10-01 16:31:26,106-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler7) [3cc45c32] VM '899e3a1c-dcc4-4426-9dd4-0f1e8b94a2b8'(ipa) moved from 'WaitForLaunch' --> 'PoweringUp' 2017-10-01 16:31:26,111-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVDSCommand] (DefaultQuartzScheduler7) [3cc45c32] START, FullListVDSCommand(HostName = virt2, FullListVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vmIds='[899e3a1c-dcc4-4426-9dd4-0f1e8b94a2b8]'}), log id: 7507eb95 2017-10-01 16:31:26,125-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-6-thread-44) [7c32dd71-7364-4968-b061-bfc1546ad1ed] FINISH, CreateVDSCommand, log id: 4f99e3b 2017-10-01 16:31:26,128-04 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-6-thread-44) [7c32dd71-7364-4968-b061-bfc1546ad1ed] FINISH, CreateVmVDSCommand, return: WaitForLaunch, log id: 2077842b 2017-10-01 16:31:26,129-04 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-6-thread-44) [7c32dd71-7364-4968-b061-bfc1546ad1ed] Lock freed to object 'EngineLock:{exclusiveLocks='[5885c43a-b133-45dc-b8d5-1aa2a4a19579=VM]', sharedLocks=''}' 2017-10-01 16:31:26,139-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-44) [7c32dd71-7364-4968-b061-bfc1546ad1ed] EVENT_ID: USER_STARTED_VM(153), Correlation ID: 7c32dd71-7364-4968-b061-bfc1546ad1ed, Job ID: b6419254-9ac4-4999-a933-824428f2b040, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM prs was started by admin@internal (Host: virt3). 2017-10-01 16:31:26,247-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVDSCommand] (DefaultQuartzScheduler7) [3cc45c32] FINISH, FullListVDSCommand, return: [{acpiEnable=true, emulatedMachine=pc-i440fx-rhel7.3.0, vmId=899e3a1c-dcc4-4426-9dd4-0f1e8b94a2b8, guestDiskMapping={}, transparentHugePages=true, timeOffset=0, cpuType=SandyBridge, smp=2, pauseCode=NOERR, guestNumaNodes=[Ljava.lang.Object;@69dfb92d, smartcardEnable=false, custom={device_c4ca6d01-88f1-4cd5-ba37-ce543be95f2fdevice_bf4bfb99-4556-4b45-af2b-38af29ab24e5device_8a92516a-5882-4ea5-96c2-1e1f7d6c95d5device_69c8a952-095f-4458-b7e9-794af0d7ad68=VmDevice:{id='VmDeviceId:{deviceId='69c8a952-095f-4458-b7e9-794af0d7ad68', vmId='899e3a1c-dcc4-4426-9dd4-0f1e8b94a2b8'}', device='ide', type='CONTROLLER', bootOrder='0', specParams='[]', address='{slot=0x01, bus=0x00, domain=0x0000, type=pci, function=0x1}', managed='false', plugged='true', readOnly='false', deviceAlias='ide', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_c4ca6d01-88f1-4cd5-ba37-ce543be95f2fdevice_bf4bfb99-4556-4b45-af2b-38af29ab24e5=VmDevice:{id='VmDeviceId:{deviceId='bf4bfb99-4556-4b45-af2b-38af29ab24e5', vmId='899e3a1c-dcc4-4426-9dd4-0f1e8b94a2b8'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=2}', managed='false', plugged='true', readOnly='false', deviceAlias='channel1', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_c4ca6d01-88f1-4cd5-ba37-ce543be95f2fdevice_bf4bfb99-4556-4b45-af2b-38af29ab24e5device_8a92516a-5882-4ea5-96c2-1e1f7d6c95d5=VmDevice:{id='VmDeviceId:{deviceId='8a92516a-5882-4ea5-96c2-1e1f7d6c95d5', vmId='899e3a1c-dcc4-4426-9dd4-0f1e8b94a2b8'}', device='spicevmc', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=3}', managed='false', plugged='true', readOnly='false', deviceAlias='channel2', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_c4ca6d01-88f1-4cd5-ba37-ce543be95f2f=VmDevice:{id='VmDeviceId:{deviceId='c4ca6d01-88f1-4cd5-ba37-ce543be95f2f', vmId='899e3a1c-dcc4-4426-9dd4-0f1e8b94a2b8'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=1}', managed='false', plugged='true', readOnly='false', deviceAlias='channel0', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}}, vmType=kvm, memSize=4096, smpCoresPerSocket=1, vmName=ipa, nice=0, status=Up, maxMemSize=16384, bootMenuEnable=false, pid=2467, smpThreadsPerCore=1, memGuaranteedSize=2730, kvmEnable=true, displayNetwork=yorku, devices=[Ljava.lang.Object;@77cfc445, display=qxl, maxVCpus=16, clientIp=, statusTime=4726569350, maxMemSlots=16}], log id: 7507eb95 2017-10-01 16:31:26,248-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler7) [3cc45c32] Received a spice Device without an address when processing VM 899e3a1c-dcc4-4426-9dd4-0f1e8b94a2b8 devices, skipping device: {device=spice, specParams={fileTransferEnable=true, displayNetwork=yorku, copyPasteEnable=true, displayIp=130.63.94.35, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,ssmartcard,susbredir}, type=graphics, deviceId=00000023-0023-0023-0023-00000000016f, tlsPort=5900} 2017-10-01 16:31:28,167-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmsStatisticsFetcher] (DefaultQuartzScheduler2) [5cd61352] Fetched 1 VMs from VDS '5ae29ed2-9c0f-4754-8df3-b43d14726846' 2017-10-01 16:31:28,167-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler2) [5cd61352] VM 'db3e08fb-c111-442d-aeec-1507da30f486'(print) moved from 'WaitForLaunch' --> 'PoweringUp' 2017-10-01 16:31:28,173-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVDSCommand] (DefaultQuartzScheduler2) [5cd61352] START, FullListVDSCommand(HostName = virt4, FullListVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vmIds='[db3e08fb-c111-442d-aeec-1507da30f486]'}), log id: 25e2b6ae 2017-10-01 16:31:29,183-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVDSCommand] (DefaultQuartzScheduler2) [5cd61352] FINISH, FullListVDSCommand, return: [{acpiEnable=true, emulatedMachine=pc-i440fx-rhel7.3.0, vmId=db3e08fb-c111-442d-aeec-1507da30f486, guestDiskMapping={}, transparentHugePages=true, timeOffset=0, cpuType=SandyBridge, smp=2, pauseCode=NOERR, guestNumaNodes=[Ljava.lang.Object;@314a0cea, smartcardEnable=false, custom={device_547649a9-4bf9-49b8-91a4-12dfb46dab53=VmDevice:{id='VmDeviceId:{deviceId='547649a9-4bf9-49b8-91a4-12dfb46dab53', vmId='db3e08fb-c111-442d-aeec-1507da30f486'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=1}', managed='false', plugged='true', readOnly='false', deviceAlias='channel0', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_547649a9-4bf9-49b8-91a4-12dfb46dab53device_b8489d3b-06ad-4649-b6af-26f137b2b702device_7bf8fe87-fe28-47a4-907c-b0665ad11d14device_296d9682-1520-452e-974e-15e1b8a69dd6=VmDevice:{id='VmDeviceId:{deviceId='296d9682-1520-452e-974e-15e1b8a69dd6', vmId='db3e08fb-c111-442d-aeec-1507da30f486'}', device='ide', type='CONTROLLER', bootOrder='0', specParams='[]', address='{slot=0x01, bus=0x00, domain=0x0000, type=pci, function=0x1}', managed='false', plugged='true', readOnly='false', deviceAlias='ide', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_547649a9-4bf9-49b8-91a4-12dfb46dab53device_b8489d3b-06ad-4649-b6af-26f137b2b702=VmDevice:{id='VmDeviceId:{deviceId='b8489d3b-06ad-4649-b6af-26f137b2b702', vmId='db3e08fb-c111-442d-aeec-1507da30f486'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=2}', managed='false', plugged='true', readOnly='false', deviceAlias='channel1', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_547649a9-4bf9-49b8-91a4-12dfb46dab53device_b8489d3b-06ad-4649-b6af-26f137b2b702device_7bf8fe87-fe28-47a4-907c-b0665ad11d14=VmDevice:{id='VmDeviceId:{deviceId='7bf8fe87-fe28-47a4-907c-b0665ad11d14', vmId='db3e08fb-c111-442d-aeec-1507da30f486'}', device='spicevmc', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=3}', managed='false', plugged='true', readOnly='false', deviceAlias='channel2', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}}, vmType=kvm, memSize=4096, smpCoresPerSocket=1, vmName=print, nice=0, status=Up, maxMemSize=16384, bootMenuEnable=false, pid=107004, smpThreadsPerCore=1, memGuaranteedSize=4096, kvmEnable=true, displayNetwork=yorku, devices=[Ljava.lang.Object;@9c1411c, display=qxl, maxVCpus=16, clientIp=, statusTime=5257192850, maxMemSlots=16}], log id: 25e2b6ae 2017-10-01 16:31:29,184-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler2) [5cd61352] Received a memballoon Device without an address when processing VM db3e08fb-c111-442d-aeec-1507da30f486 devices, skipping device: {device=memballoon, specParams={model=none}, type=balloon, alias=balloon0} 2017-10-01 16:31:29,184-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler2) [5cd61352] Received a spice Device without an address when processing VM db3e08fb-c111-442d-aeec-1507da30f486 devices, skipping device: {device=spice, specParams={fileTransferEnable=true, displayNetwork=yorku, copyPasteEnable=true, displayIp=130.63.94.37, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,ssmartcard,susbredir}, type=graphics, deviceId=00000035-0035-0035-0035-0000000003a5, tlsPort=5900} 2017-10-01 16:31:34,358-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmsStatisticsFetcher] (DefaultQuartzScheduler8) [64361bf] Fetched 2 VMs from VDS 'e1e71e9d-a0cd-47db-96c6-f724f6cea736' 2017-10-01 16:31:34,359-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler8) [64361bf] VM '5885c43a-b133-45dc-b8d5-1aa2a4a19579'(prs) moved from 'WaitForLaunch' --> 'PoweringUp' 2017-10-01 16:31:34,365-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVDSCommand] (DefaultQuartzScheduler8) [64361bf] START, FullListVDSCommand(HostName = virt3, FullListVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vmIds='[5885c43a-b133-45dc-b8d5-1aa2a4a19579]'}), log id: 89bd4e3 2017-10-01 16:31:35,378-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVDSCommand] (DefaultQuartzScheduler8) [64361bf] FINISH, FullListVDSCommand, return: [{acpiEnable=true, emulatedMachine=pc-i440fx-rhel7.3.0, vmId=5885c43a-b133-45dc-b8d5-1aa2a4a19579, guestDiskMapping={}, transparentHugePages=true, timeOffset=0, cpuType=SandyBridge, smp=1, pauseCode=NOERR, guestNumaNodes=[Ljava.lang.Object;@22df6bea, smartcardEnable=false, custom={device_83673bc9-c843-4588-854d-5166d505dbba=VmDevice:{id='VmDeviceId:{deviceId='83673bc9-c843-4588-854d-5166d505dbba', vmId='5885c43a-b133-45dc-b8d5-1aa2a4a19579'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=1}', managed='false', plugged='true', readOnly='false', deviceAlias='channel0', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_83673bc9-c843-4588-854d-5166d505dbbadevice_9494c9a1-5388-4595-bdac-a2cdfe926c8ddevice_d8c69979-7261-4dd5-b667-01d2e92187f4device_1f01430f-3702-4ec6-83e1-fdb9a730f67e=VmDevice:{id='VmDeviceId:{deviceId='1f01430f-3702-4ec6-83e1-fdb9a730f67e', vmId='5885c43a-b133-45dc-b8d5-1aa2a4a19579'}', device='ide', type='CONTROLLER', bootOrder='0', specParams='[]', address='{slot=0x01, bus=0x00, domain=0x0000, type=pci, function=0x1}', managed='false', plugged='true', readOnly='false', deviceAlias='ide', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_83673bc9-c843-4588-854d-5166d505dbbadevice_9494c9a1-5388-4595-bdac-a2cdfe926c8ddevice_d8c69979-7261-4dd5-b667-01d2e92187f4=VmDevice:{id='VmDeviceId:{deviceId='d8c69979-7261-4dd5-b667-01d2e92187f4', vmId='5885c43a-b133-45dc-b8d5-1aa2a4a19579'}', device='spicevmc', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=3}', managed='false', plugged='true', readOnly='false', deviceAlias='channel2', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_83673bc9-c843-4588-854d-5166d505dbbadevice_9494c9a1-5388-4595-bdac-a2cdfe926c8d=VmDevice:{id='VmDeviceId:{deviceId='9494c9a1-5388-4595-bdac-a2cdfe926c8d', vmId='5885c43a-b133-45dc-b8d5-1aa2a4a19579'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=2}', managed='false', plugged='true', readOnly='false', deviceAlias='channel1', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}}, vmType=kvm, memSize=1024, smpCoresPerSocket=1, vmName=prs, nice=0, status=Up, maxMemSize=4096, bootMenuEnable=false, pid=6630, smpThreadsPerCore=1, memGuaranteedSize=682, kvmEnable=true, displayNetwork=yorku, devices=[Ljava.lang.Object;@6b7aed95, display=qxl, maxVCpus=16, clientIp=, statusTime=5262510820, maxMemSlots=16}], log id: 89bd4e3 2017-10-01 16:31:35,381-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler8) [64361bf] Received a memballoon Device without an address when processing VM 5885c43a-b133-45dc-b8d5-1aa2a4a19579 devices, skipping device: {device=memballoon, specParams={model=none}, type=balloon, alias=balloon0} 2017-10-01 16:31:35,382-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler8) [64361bf] Received a spice Device without an address when processing VM 5885c43a-b133-45dc-b8d5-1aa2a4a19579 devices, skipping device: {device=spice, specParams={fileTransferEnable=true, displayNetwork=yorku, copyPasteEnable=true, displayIp=130.63.94.36, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,ssmartcard,susbredir}, type=graphics, deviceId=8c3d4408-d4ef-44d2-9442-b048e1668bee, tlsPort=5901} 2017-10-01 16:31:56,904-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [2a342d4] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 16:32:06,216-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler6) [4d9923dc] VM 'a8d4e149-55ec-4961-a414-d7d1e1b05a59'(fix) moved from 'PoweringUp' --> 'Up' 2017-10-01 16:32:06,225-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [4d9923dc] EVENT_ID: USER_RUN_VM(32), Correlation ID: 509ee821-b2e0-4b7a-9157-2901b521595b, Job ID: abfaf0eb-ad65-45f0-b347-68799bd75bb9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM fix started on Host virt1 2017-10-01 16:32:28,150-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler2) [5d44feb0] VM '899e3a1c-dcc4-4426-9dd4-0f1e8b94a2b8'(ipa) moved from 'PoweringUp' --> 'Up' 2017-10-01 16:32:28,158-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [5d44feb0] EVENT_ID: USER_RUN_VM(32), Correlation ID: 339336f6-ec1b-4d8c-b747-75194de8f838, Job ID: 29634791-24f1-4105-a1fd-96faa53658da, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM ipa started on Host virt2 2017-10-01 16:32:33,223-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler8) [37e01eee] VM 'db3e08fb-c111-442d-aeec-1507da30f486'(print) moved from 'PoweringUp' --> 'Up' 2017-10-01 16:32:33,230-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [37e01eee] EVENT_ID: USER_RUN_VM(32), Correlation ID: a581bf39-0479-4cab-b2b7-0a39de2024cf, Job ID: c308d1c2-d2d6-4237-80c0-0532d52c100a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM print started on Host virt4 2017-10-01 16:32:38,485-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler3) [4c45909a] VM '5885c43a-b133-45dc-b8d5-1aa2a4a19579'(prs) moved from 'PoweringUp' --> 'Up' 2017-10-01 16:32:38,492-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [4c45909a] EVENT_ID: USER_RUN_VM(32), Correlation ID: 7c32dd71-7364-4968-b061-bfc1546ad1ed, Job ID: b6419254-9ac4-4999-a933-824428f2b040, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM prs started on Host virt3 2017-10-01 16:32:56,928-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [1330fa8a] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 16:33:40,078-04 INFO [org.ovirt.engine.core.sso.utils.AuthenticationUtils] (default task-201) [] User jas@eecs.yorku.ca successfully logged in with scopes: ovirt-app-admin ovirt-app-api ovirt-app-portal ovirt-ext=auth:sequence-priority=~ ovirt-ext=revoke:revoke-all ovirt-ext=token-info:authz-search ovirt-ext=token-info:public-authz-search ovirt-ext=token-info:validate ovirt-ext=token:password-access 2017-10-01 16:33:40,174-04 INFO [org.ovirt.engine.core.bll.aaa.CreateUserSessionCommand] (default task-202) [5044f661] Running command: CreateUserSessionCommand internal: false. 2017-10-01 16:33:40,182-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-202) [5044f661] EVENT_ID: USER_VDC_LOGIN(30), Correlation ID: 5044f661, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: User jas@eecs.yorku.ca-authz logged in. 2017-10-01 16:33:43,274-04 ERROR [org.ovirt.engine.core.utils.servlet.ServletUtils] (default task-210) [] Can't read file '/usr/share/ovirt-engine/files/spice/SpiceVersion.txt' for request '/ovirt-engine/services/files/spice/SpiceVersion.txt', will send a 404 error response. 2017-10-01 16:33:56,945-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [669d973b] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 16:34:53,555-04 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-6-thread-19) [] Executing a command 'java.util.concurrent.FutureTask', but note that there are 9 tasks in the queue. 2017-10-01 16:34:53,555-04 INFO [org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner] (org.ovirt.thread.pool-6-thread-19) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] Start running Validate for command number 1/10 (Command type 'RunVm') 2017-10-01 16:34:53,555-04 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-6-thread-17) [] Executing a command 'java.util.concurrent.FutureTask', but note that there are 8 tasks in the queue. 2017-10-01 16:34:53,555-04 INFO [org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner] (org.ovirt.thread.pool-6-thread-17) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] Start running Validate for command number 2/10 (Command type 'RunVm') 2017-10-01 16:34:53,555-04 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-6-thread-1) [] Executing a command 'java.util.concurrent.FutureTask', but note that there are 7 tasks in the queue. 2017-10-01 16:34:53,555-04 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-6-thread-38) [] Executing a command 'java.util.concurrent.FutureTask', but note that there are 6 tasks in the queue. 2017-10-01 16:34:53,555-04 INFO [org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner] (org.ovirt.thread.pool-6-thread-38) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] Start running Validate for command number 4/10 (Command type 'RunVm') 2017-10-01 16:34:53,555-04 INFO [org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner] (org.ovirt.thread.pool-6-thread-1) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] Start running Validate for command number 3/10 (Command type 'RunVm') 2017-10-01 16:34:53,555-04 INFO [org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner] (org.ovirt.thread.pool-6-thread-12) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] Start running Validate for command number 6/10 (Command type 'RunVm') 2017-10-01 16:34:53,555-04 INFO [org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner] (org.ovirt.thread.pool-6-thread-16) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] Start running Validate for command number 5/10 (Command type 'RunVm') 2017-10-01 16:34:53,555-04 INFO [org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner] (org.ovirt.thread.pool-6-thread-43) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] Start running Validate for command number 7/10 (Command type 'RunVm') 2017-10-01 16:34:53,555-04 INFO [org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner] (org.ovirt.thread.pool-6-thread-20) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] Start running Validate for command number 8/10 (Command type 'RunVm') 2017-10-01 16:34:53,555-04 INFO [org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner] (org.ovirt.thread.pool-6-thread-4) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] Start running Validate for command number 9/10 (Command type 'RunVm') 2017-10-01 16:34:53,555-04 INFO [org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner] (org.ovirt.thread.pool-6-thread-31) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] Start running Validate for command number 10/10 (Command type 'RunVm') 2017-10-01 16:34:53,556-04 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-6-thread-38) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] Lock Acquired to object 'EngineLock:{exclusiveLocks='[911d6883-607c-464e-b46b-8e6141bdd2de=VM]', sharedLocks=''}' 2017-10-01 16:34:53,556-04 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-6-thread-17) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] Lock Acquired to object 'EngineLock:{exclusiveLocks='[ae56c31c-71f7-433a-b198-a1c2c846325e=VM]', sharedLocks=''}' 2017-10-01 16:34:53,556-04 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-6-thread-19) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] Lock Acquired to object 'EngineLock:{exclusiveLocks='[1325cebf-834e-4571-a3e1-82beb0021ffe=VM]', sharedLocks=''}' 2017-10-01 16:34:53,557-04 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-6-thread-1) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] Lock Acquired to object 'EngineLock:{exclusiveLocks='[aa487207-7ff4-465a-9d9b-2a103d50dc77=VM]', sharedLocks=''}' 2017-10-01 16:34:53,557-04 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-6-thread-12) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] Lock Acquired to object 'EngineLock:{exclusiveLocks='[cc1dbe93-bb80-451e-8b3b-c9cc5e90432f=VM]', sharedLocks=''}' 2017-10-01 16:34:53,557-04 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-6-thread-31) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] Lock Acquired to object 'EngineLock:{exclusiveLocks='[07331c78-6e09-41b4-9b29-edda32f46351=VM]', sharedLocks=''}' 2017-10-01 16:34:53,557-04 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-6-thread-43) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] Lock Acquired to object 'EngineLock:{exclusiveLocks='[bfe9a524-2238-4bab-a91f-5dc6ed270896=VM]', sharedLocks=''}' 2017-10-01 16:34:53,557-04 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-6-thread-4) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] Lock Acquired to object 'EngineLock:{exclusiveLocks='[9ea2821e-557f-45cb-9872-1e6c5aeaf246=VM]', sharedLocks=''}' 2017-10-01 16:34:53,557-04 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-6-thread-20) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] Lock Acquired to object 'EngineLock:{exclusiveLocks='[c64d80f9-a3bd-44f3-89c4-f1b9b509cdde=VM]', sharedLocks=''}' 2017-10-01 16:34:53,562-04 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-6-thread-16) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] Lock Acquired to object 'EngineLock:{exclusiveLocks='[a82b4cc3-6ef6-4a8e-97ef-1d4ea1b7e07c=VM]', sharedLocks=''}' 2017-10-01 16:34:53,579-04 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (org.ovirt.thread.pool-6-thread-17) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] START, IsVmDuringInitiatingVDSCommand( IsVmDuringInitiatingVDSCommandParameters:{runAsync='true', vmId='ae56c31c-71f7-433a-b198-a1c2c846325e'}), log id: 6a92bba4 2017-10-01 16:34:53,580-04 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (org.ovirt.thread.pool-6-thread-17) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 6a92bba4 2017-10-01 16:34:53,598-04 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (org.ovirt.thread.pool-6-thread-43) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] START, IsVmDuringInitiatingVDSCommand( IsVmDuringInitiatingVDSCommandParameters:{runAsync='true', vmId='bfe9a524-2238-4bab-a91f-5dc6ed270896'}), log id: 59435422 2017-10-01 16:34:53,598-04 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (org.ovirt.thread.pool-6-thread-43) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 59435422 2017-10-01 16:34:53,620-04 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (org.ovirt.thread.pool-6-thread-19) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] START, IsVmDuringInitiatingVDSCommand( IsVmDuringInitiatingVDSCommandParameters:{runAsync='true', vmId='1325cebf-834e-4571-a3e1-82beb0021ffe'}), log id: a2e46e2 2017-10-01 16:34:53,620-04 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (org.ovirt.thread.pool-6-thread-19) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: a2e46e2 2017-10-01 16:34:53,620-04 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (org.ovirt.thread.pool-6-thread-31) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] START, IsVmDuringInitiatingVDSCommand( IsVmDuringInitiatingVDSCommandParameters:{runAsync='true', vmId='07331c78-6e09-41b4-9b29-edda32f46351'}), log id: 47a1c448 2017-10-01 16:34:53,621-04 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (org.ovirt.thread.pool-6-thread-20) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] START, IsVmDuringInitiatingVDSCommand( IsVmDuringInitiatingVDSCommandParameters:{runAsync='true', vmId='c64d80f9-a3bd-44f3-89c4-f1b9b509cdde'}), log id: 5e9d9588 2017-10-01 16:34:53,621-04 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (org.ovirt.thread.pool-6-thread-31) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 47a1c448 2017-10-01 16:34:53,621-04 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (org.ovirt.thread.pool-6-thread-20) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 5e9d9588 2017-10-01 16:34:53,651-04 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (org.ovirt.thread.pool-6-thread-38) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] START, IsVmDuringInitiatingVDSCommand( IsVmDuringInitiatingVDSCommandParameters:{runAsync='true', vmId='911d6883-607c-464e-b46b-8e6141bdd2de'}), log id: 7cb9faa0 2017-10-01 16:34:53,651-04 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (org.ovirt.thread.pool-6-thread-38) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 7cb9faa0 2017-10-01 16:34:53,668-04 INFO [org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner] (org.ovirt.thread.pool-6-thread-17) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] Finish handling Validate for command number 2/10 (Command type 'RunVm') 2017-10-01 16:34:53,669-04 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (org.ovirt.thread.pool-6-thread-16) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] START, IsVmDuringInitiatingVDSCommand( IsVmDuringInitiatingVDSCommandParameters:{runAsync='true', vmId='a82b4cc3-6ef6-4a8e-97ef-1d4ea1b7e07c'}), log id: 999e10a 2017-10-01 16:34:53,669-04 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (org.ovirt.thread.pool-6-thread-16) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 999e10a 2017-10-01 16:34:53,677-04 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (org.ovirt.thread.pool-6-thread-1) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] START, IsVmDuringInitiatingVDSCommand( IsVmDuringInitiatingVDSCommandParameters:{runAsync='true', vmId='aa487207-7ff4-465a-9d9b-2a103d50dc77'}), log id: 1b5742c4 2017-10-01 16:34:53,677-04 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (org.ovirt.thread.pool-6-thread-1) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 1b5742c4 2017-10-01 16:34:53,677-04 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (org.ovirt.thread.pool-6-thread-4) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] START, IsVmDuringInitiatingVDSCommand( IsVmDuringInitiatingVDSCommandParameters:{runAsync='true', vmId='9ea2821e-557f-45cb-9872-1e6c5aeaf246'}), log id: 4881189e 2017-10-01 16:34:53,678-04 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (org.ovirt.thread.pool-6-thread-4) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 4881189e 2017-10-01 16:34:53,681-04 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (org.ovirt.thread.pool-6-thread-12) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] START, IsVmDuringInitiatingVDSCommand( IsVmDuringInitiatingVDSCommandParameters:{runAsync='true', vmId='cc1dbe93-bb80-451e-8b3b-c9cc5e90432f'}), log id: 3d5c88c1 2017-10-01 16:34:53,681-04 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (org.ovirt.thread.pool-6-thread-12) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 3d5c88c1 2017-10-01 16:34:53,706-04 INFO [org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner] (org.ovirt.thread.pool-6-thread-43) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] Finish handling Validate for command number 7/10 (Command type 'RunVm') 2017-10-01 16:34:53,708-04 INFO [org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner] (org.ovirt.thread.pool-6-thread-19) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] Finish handling Validate for command number 1/10 (Command type 'RunVm') 2017-10-01 16:34:53,715-04 INFO [org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner] (org.ovirt.thread.pool-6-thread-20) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] Finish handling Validate for command number 8/10 (Command type 'RunVm') 2017-10-01 16:34:53,719-04 INFO [org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner] (org.ovirt.thread.pool-6-thread-31) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] Finish handling Validate for command number 10/10 (Command type 'RunVm') 2017-10-01 16:34:53,741-04 INFO [org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner] (org.ovirt.thread.pool-6-thread-38) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] Finish handling Validate for command number 4/10 (Command type 'RunVm') 2017-10-01 16:34:53,744-04 INFO [org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner] (org.ovirt.thread.pool-6-thread-1) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] Finish handling Validate for command number 3/10 (Command type 'RunVm') 2017-10-01 16:34:53,745-04 INFO [org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner] (org.ovirt.thread.pool-6-thread-16) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] Finish handling Validate for command number 5/10 (Command type 'RunVm') 2017-10-01 16:34:53,768-04 INFO [org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner] (org.ovirt.thread.pool-6-thread-4) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] Finish handling Validate for command number 9/10 (Command type 'RunVm') 2017-10-01 16:34:53,773-04 INFO [org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner] (org.ovirt.thread.pool-6-thread-12) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] Finish handling Validate for command number 6/10 (Command type 'RunVm') 2017-10-01 16:34:53,773-04 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-6-thread-30) [] Executing a command 'java.util.concurrent.FutureTask', but note that there are 9 tasks in the queue. 2017-10-01 16:34:53,773-04 INFO [org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner] (org.ovirt.thread.pool-6-thread-30) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] Start running Validate for command number 11/20 (Command type 'RunVm') 2017-10-01 16:34:53,773-04 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-6-thread-45) [] Executing a command 'java.util.concurrent.FutureTask', but note that there are 8 tasks in the queue. 2017-10-01 16:34:53,774-04 INFO [org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner] (org.ovirt.thread.pool-6-thread-45) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] Start running Validate for command number 12/20 (Command type 'RunVm') 2017-10-01 16:34:53,774-04 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-6-thread-21) [] Executing a command 'java.util.concurrent.FutureTask', but note that there are 7 tasks in the queue. 2017-10-01 16:34:53,774-04 WARN [org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil] (org.ovirt.thread.pool-6-thread-39) [] Executing a command 'java.util.concurrent.FutureTask', but note that there are 6 tasks in the queue. 2017-10-01 16:34:53,774-04 INFO [org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner] (org.ovirt.thread.pool-6-thread-21) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] Start running Validate for command number 13/20 (Command type 'RunVm') 2017-10-01 16:34:53,774-04 INFO [org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner] (org.ovirt.thread.pool-6-thread-39) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] Start running Validate for command number 14/20 (Command type 'RunVm') 2017-10-01 16:34:53,774-04 INFO [org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner] (org.ovirt.thread.pool-6-thread-49) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] Start running Validate for command number 15/20 (Command type 'RunVm') 2017-10-01 16:34:53,774-04 INFO [org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner] (org.ovirt.thread.pool-6-thread-22) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] Start running Validate for command number 16/20 (Command type 'RunVm') 2017-10-01 16:34:53,774-04 INFO [org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner] (org.ovirt.thread.pool-6-thread-35) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] Start running Validate for command number 17/20 (Command type 'RunVm') 2017-10-01 16:34:53,774-04 INFO [org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner] (org.ovirt.thread.pool-6-thread-37) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] Start running Validate for command number 18/20 (Command type 'RunVm') 2017-10-01 16:34:53,774-04 INFO [org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner] (org.ovirt.thread.pool-6-thread-6) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] Start running Validate for command number 19/20 (Command type 'RunVm') 2017-10-01 16:34:53,774-04 INFO [org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner] (org.ovirt.thread.pool-6-thread-18) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] Start running Validate for command number 20/20 (Command type 'RunVm') 2017-10-01 16:34:53,781-04 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-6-thread-49) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] Lock Acquired to object 'EngineLock:{exclusiveLocks='[f87ec08d-0fdf-42ba-8aac-c66c404d7aae=VM]', sharedLocks=''}' 2017-10-01 16:34:53,781-04 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-6-thread-30) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] Lock Acquired to object 'EngineLock:{exclusiveLocks='[d56363ad-9cb1-4a70-a33a-ce439b044d7d=VM]', sharedLocks=''}' 2017-10-01 16:34:53,781-04 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-6-thread-45) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] Lock Acquired to object 'EngineLock:{exclusiveLocks='[1c7e966f-29e8-4788-b64c-86e52f415061=VM]', sharedLocks=''}' 2017-10-01 16:34:53,781-04 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-6-thread-39) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] Lock Acquired to object 'EngineLock:{exclusiveLocks='[ae65a104-ed3f-40e0-ba92-5708cf338918=VM]', sharedLocks=''}' 2017-10-01 16:34:53,781-04 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-6-thread-35) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] Lock Acquired to object 'EngineLock:{exclusiveLocks='[1734fed5-fc97-4714-bb63-55e8f4eb6d34=VM]', sharedLocks=''}' 2017-10-01 16:34:53,781-04 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-6-thread-21) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] Lock Acquired to object 'EngineLock:{exclusiveLocks='[59287796-c634-4f3c-94b1-2c9699986637=VM]', sharedLocks=''}' 2017-10-01 16:34:53,782-04 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-6-thread-18) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] Lock Acquired to object 'EngineLock:{exclusiveLocks='[3bb83ad5-dc00-445d-8708-2db18c53e0e6=VM]', sharedLocks=''}' 2017-10-01 16:34:53,782-04 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-6-thread-6) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] Lock Acquired to object 'EngineLock:{exclusiveLocks='[d66235b4-7c8d-4da2-a572-82ecc9c53a59=VM]', sharedLocks=''}' 2017-10-01 16:34:53,782-04 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-6-thread-37) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] Lock Acquired to object 'EngineLock:{exclusiveLocks='[3e5a5728-8940-4fe7-b4af-11fb8e12f116=VM]', sharedLocks=''}' 2017-10-01 16:34:53,793-04 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-6-thread-22) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] Lock Acquired to object 'EngineLock:{exclusiveLocks='[64734fca-4d6c-443c-82d1-abf7cf106601=VM]', sharedLocks=''}' 2017-10-01 16:34:53,821-04 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (org.ovirt.thread.pool-6-thread-35) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] START, IsVmDuringInitiatingVDSCommand( IsVmDuringInitiatingVDSCommandParameters:{runAsync='true', vmId='1734fed5-fc97-4714-bb63-55e8f4eb6d34'}), log id: 6b2a891 2017-10-01 16:34:53,821-04 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (org.ovirt.thread.pool-6-thread-35) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 6b2a891 2017-10-01 16:34:53,827-04 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (org.ovirt.thread.pool-6-thread-6) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] START, IsVmDuringInitiatingVDSCommand( IsVmDuringInitiatingVDSCommandParameters:{runAsync='true', vmId='d66235b4-7c8d-4da2-a572-82ecc9c53a59'}), log id: 72c3c704 2017-10-01 16:34:53,827-04 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (org.ovirt.thread.pool-6-thread-6) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 72c3c704 2017-10-01 16:34:53,828-04 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (org.ovirt.thread.pool-6-thread-37) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] START, IsVmDuringInitiatingVDSCommand( IsVmDuringInitiatingVDSCommandParameters:{runAsync='true', vmId='3e5a5728-8940-4fe7-b4af-11fb8e12f116'}), log id: 1f5fc2e9 2017-10-01 16:34:53,828-04 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (org.ovirt.thread.pool-6-thread-37) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 1f5fc2e9 2017-10-01 16:34:53,829-04 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (org.ovirt.thread.pool-6-thread-21) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] START, IsVmDuringInitiatingVDSCommand( IsVmDuringInitiatingVDSCommandParameters:{runAsync='true', vmId='59287796-c634-4f3c-94b1-2c9699986637'}), log id: 1a58d0b 2017-10-01 16:34:53,829-04 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (org.ovirt.thread.pool-6-thread-21) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 1a58d0b 2017-10-01 16:34:53,840-04 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (org.ovirt.thread.pool-6-thread-45) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] START, IsVmDuringInitiatingVDSCommand( IsVmDuringInitiatingVDSCommandParameters:{runAsync='true', vmId='1c7e966f-29e8-4788-b64c-86e52f415061'}), log id: 29578c8c 2017-10-01 16:34:53,840-04 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (org.ovirt.thread.pool-6-thread-45) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 29578c8c 2017-10-01 16:34:53,848-04 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (org.ovirt.thread.pool-6-thread-18) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] START, IsVmDuringInitiatingVDSCommand( IsVmDuringInitiatingVDSCommandParameters:{runAsync='true', vmId='3bb83ad5-dc00-445d-8708-2db18c53e0e6'}), log id: 3f311a2a 2017-10-01 16:34:53,848-04 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (org.ovirt.thread.pool-6-thread-18) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 3f311a2a 2017-10-01 16:34:53,850-04 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (org.ovirt.thread.pool-6-thread-30) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] START, IsVmDuringInitiatingVDSCommand( IsVmDuringInitiatingVDSCommandParameters:{runAsync='true', vmId='d56363ad-9cb1-4a70-a33a-ce439b044d7d'}), log id: 3af8991c 2017-10-01 16:34:53,850-04 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (org.ovirt.thread.pool-6-thread-30) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 3af8991c 2017-10-01 16:34:53,850-04 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (org.ovirt.thread.pool-6-thread-22) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] START, IsVmDuringInitiatingVDSCommand( IsVmDuringInitiatingVDSCommandParameters:{runAsync='true', vmId='64734fca-4d6c-443c-82d1-abf7cf106601'}), log id: 99afd8b 2017-10-01 16:34:53,850-04 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (org.ovirt.thread.pool-6-thread-22) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 99afd8b 2017-10-01 16:34:53,851-04 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (org.ovirt.thread.pool-6-thread-39) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] START, IsVmDuringInitiatingVDSCommand( IsVmDuringInitiatingVDSCommandParameters:{runAsync='true', vmId='ae65a104-ed3f-40e0-ba92-5708cf338918'}), log id: 2ff7bd2e 2017-10-01 16:34:53,851-04 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (org.ovirt.thread.pool-6-thread-39) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 2ff7bd2e 2017-10-01 16:34:53,853-04 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (org.ovirt.thread.pool-6-thread-49) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] START, IsVmDuringInitiatingVDSCommand( IsVmDuringInitiatingVDSCommandParameters:{runAsync='true', vmId='f87ec08d-0fdf-42ba-8aac-c66c404d7aae'}), log id: 4233bc1c 2017-10-01 16:34:53,853-04 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (org.ovirt.thread.pool-6-thread-49) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 4233bc1c 2017-10-01 16:34:53,904-04 INFO [org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner] (org.ovirt.thread.pool-6-thread-35) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] Finish handling Validate for command number 17/20 (Command type 'RunVm') 2017-10-01 16:34:53,912-04 INFO [org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner] (org.ovirt.thread.pool-6-thread-37) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] Finish handling Validate for command number 18/20 (Command type 'RunVm') 2017-10-01 16:34:53,922-04 INFO [org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner] (org.ovirt.thread.pool-6-thread-45) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] Finish handling Validate for command number 12/20 (Command type 'RunVm') 2017-10-01 16:34:53,924-04 INFO [org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner] (org.ovirt.thread.pool-6-thread-18) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] Finish handling Validate for command number 20/20 (Command type 'RunVm') 2017-10-01 16:34:53,924-04 INFO [org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner] (org.ovirt.thread.pool-6-thread-6) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] Finish handling Validate for command number 19/20 (Command type 'RunVm') 2017-10-01 16:34:53,929-04 INFO [org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner] (org.ovirt.thread.pool-6-thread-30) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] Finish handling Validate for command number 11/20 (Command type 'RunVm') 2017-10-01 16:34:53,933-04 INFO [org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner] (org.ovirt.thread.pool-6-thread-39) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] Finish handling Validate for command number 14/20 (Command type 'RunVm') 2017-10-01 16:34:53,933-04 INFO [org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner] (org.ovirt.thread.pool-6-thread-22) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] Finish handling Validate for command number 16/20 (Command type 'RunVm') 2017-10-01 16:34:53,944-04 INFO [org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner] (org.ovirt.thread.pool-6-thread-21) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] Finish handling Validate for command number 13/20 (Command type 'RunVm') 2017-10-01 16:34:53,957-04 INFO [org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner] (org.ovirt.thread.pool-6-thread-49) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] Finish handling Validate for command number 15/20 (Command type 'RunVm') 2017-10-01 16:34:53,957-04 INFO [org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner] (org.ovirt.thread.pool-6-thread-33) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] Start running Validate for command number 21/21 (Command type 'RunVm') 2017-10-01 16:34:53,959-04 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-6-thread-33) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] Lock Acquired to object 'EngineLock:{exclusiveLocks='[c8682859-9e55-460a-9ef0-d03e4a85f50e=VM]', sharedLocks=''}' 2017-10-01 16:34:54,006-04 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (org.ovirt.thread.pool-6-thread-33) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] START, IsVmDuringInitiatingVDSCommand( IsVmDuringInitiatingVDSCommandParameters:{runAsync='true', vmId='c8682859-9e55-460a-9ef0-d03e4a85f50e'}), log id: 419d3a79 2017-10-01 16:34:54,006-04 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (org.ovirt.thread.pool-6-thread-33) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 419d3a79 2017-10-01 16:34:54,092-04 INFO [org.ovirt.engine.core.bll.PrevalidatingMultipleActionsRunner] (org.ovirt.thread.pool-6-thread-33) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] Finish handling Validate for command number 21/21 (Command type 'RunVm') 2017-10-01 16:34:54,110-04 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] Running command: RunVmCommand internal: false. Entities affected : ID: 1c7e966f-29e8-4788-b64c-86e52f415061 Type: VMAction group RUN_VM with role type USER 2017-10-01 16:34:54,245-04 INFO [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] START, UpdateVmDynamicDataVDSCommand( UpdateVmDynamicDataVDSCommandParameters:{runAsync='true', hostId='null', vmId='1c7e966f-29e8-4788-b64c-86e52f415061', vmDynamic='org.ovirt.engine.core.common.businessentities.VmDynamic@461fcbaa'}), log id: 6924c6f2 2017-10-01 16:34:54,247-04 INFO [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] FINISH, UpdateVmDynamicDataVDSCommand, log id: 6924c6f2 2017-10-01 16:34:54,251-04 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] START, CreateVmVDSCommand( CreateVmVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vmId='1c7e966f-29e8-4788-b64c-86e52f415061', vm='VM [udb]'}), log id: 31b98765 2017-10-01 16:34:54,254-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] START, CreateVDSCommand(HostName = virt1, CreateVmVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vmId='1c7e966f-29e8-4788-b64c-86e52f415061', vm='VM [udb]'}), log id: 7fe35141 2017-10-01 16:34:54,275-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] VM {memGuaranteedSize=4096, smpThreadsPerCore=1, cpuType=SandyBridge, vmId=1c7e966f-29e8-4788-b64c-86e52f415061, acpiEnable=true, vmType=kvm, smp=2, smpCoresPerSocket=1, emulatedMachine=pc-i440fx-rhel7.3.0, smartcardEnable=false, guestNumaNodes=[{memory=4096, cpus=0,1, nodeIndex=0}], transparentHugePages=true, displayNetwork=yorku, vmName=udb, maxVCpus=16, kvmEnable=true, devices=[{address={bus=0x00, domain=0x0000, function=0x0, slot=0x02, type=pci}, type=video, specParams={vgamem=16384, heads=1, vram=8192, ram=65536}, device=qxl, deviceId=68697560-3360-45ee-97ef-39918fe7618e}, {type=graphics, specParams={fileTransferEnable=true, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,ssmartcard,susbredir, copyPasteEnable=true}, device=spice, deviceId=05aeb875-80f2-4a28-9070-4975d8dc2b9b}, {iface=ide, shared=false, path=, address={bus=1, controller=0, unit=0, type=drive, target=0}, readonly=true, index=2, type=disk, specParams={path=}, device=cdrom, deviceId=803d0d5b-0ac1-4db9-88fd-ddca9581c69a}, {discard=false, shared=false, address={bus=0x00, domain=0x0000, function=0x0, slot=0x07, type=pci}, imageID=a36c7a86-3260-4754-a96e-99330bb0a3b8, format=raw, optional=false, type=disk, deviceId=a36c7a86-3260-4754-a96e-99330bb0a3b8, domainID=c8653dba-63ec-4447-872c-0ffcc6af9c0f, propagateErrors=off, iface=virtio, readonly=false, poolID=aca61eac-ed8a-484b-bbee-3645148f6405, volumeID=da544d10-530a-44fb-9506-bb6b82d274f1, specParams={}, device=disk}, {discard=false, shared=false, address={bus=0x00, domain=0x0000, function=0x0, slot=0x08, type=pci}, imageID=a30a99be-b7a4-4f71-9354-f6b5cd8dc2a9, format=raw, optional=false, type=disk, deviceId=a30a99be-b7a4-4f71-9354-f6b5cd8dc2a9, domainID=c8653dba-63ec-4447-872c-0ffcc6af9c0f, propagateErrors=off, iface=virtio, readonly=false, poolID=aca61eac-ed8a-484b-bbee-3645148f6405, volumeID=0044e81a-f162-4f3c-959d-7ddf0161dcfe, specParams={}, device=disk}, {filter=vdsm-no-mac-spoofing, nicModel=pv, address={bus=0x00, domain=0x0000, function=0x0, slot=0x03, type=pci}, type=interface, specParams={inbound={}, outbound={}}, device=bridge, linkActive=true, deviceId=32214dc7-8947-478f-8d06-725adc50c3ac, macAddr=00:1a:4a:32:3d:34, network=yorku}, {filter=vdsm-no-mac-spoofing, nicModel=pv, address={bus=0x00, domain=0x0000, function=0x0, slot=0x04, type=pci}, type=interface, specParams={inbound={}, outbound={}}, device=bridge, linkActive=true, deviceId=9bb52b7a-bd3c-4aaa-8578-10c508fbef8d, macAddr=00:1a:4a:32:3d:35, network=mrpriv}, {address={bus=0x00, domain=0x0000, function=0x2, slot=0x01, type=pci}, index=0, model=piix3-uhci, type=controller, specParams={}, device=usb, deviceId=96383dbe-9f7f-4d4f-b21b-e2e9707bc9af}, {address={bus=0x00, domain=0x0000, function=0x0, slot=0x05, type=pci}, index=0, model=virtio-scsi, type=controller, specParams={}, device=scsi, deviceId=771de6d1-3719-410f-9441-ed9dd3588abc}, {address={bus=0x00, domain=0x0000, function=0x0, slot=0x06, type=pci}, type=controller, specParams={}, device=virtio-serial, deviceId=9844f6dd-3206-4a18-868a-656e3cee1e7c}, {model=virtio, type=rng, specParams={source=urandom}, device=virtio, deviceId=51dc3ead-f8e1-4c8e-b923-9f1b4b7fdbbe}], custom={device_56041595-a1a1-4da5-9c82-0766a2a679e9device_caa0fc07-0456-4e86-ac52-e18524843fa8device_93d0045d-39ba-48e6-b704-3599b39f59f9device_5a114c4b-2119-4016-9a29-69f19deb09f1=VmDevice:{id='VmDeviceId:{deviceId='5a114c4b-2119-4016-9a29-69f19deb09f1', vmId='1c7e966f-29e8-4788-b64c-86e52f415061'}', device='ide', type='CONTROLLER', bootOrder='0', specParams='[]', address='{slot=0x01, bus=0x00, domain=0x0000, type=pci, function=0x1}', managed='false', plugged='true', readOnly='false', deviceAlias='ide', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_56041595-a1a1-4da5-9c82-0766a2a679e9=VmDevice:{id='VmDeviceId:{deviceId='56041595-a1a1-4da5-9c82-0766a2a679e9', vmId='1c7e966f-29e8-4788-b64c-86e52f415061'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=1}', managed='false', plugged='true', readOnly='false', deviceAlias='channel0', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_56041595-a1a1-4da5-9c82-0766a2a679e9device_caa0fc07-0456-4e86-ac52-e18524843fa8=VmDevice:{id='VmDeviceId:{deviceId='caa0fc07-0456-4e86-ac52-e18524843fa8', vmId='1c7e966f-29e8-4788-b64c-86e52f415061'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=2}', managed='false', plugged='true', readOnly='false', deviceAlias='channel1', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_56041595-a1a1-4da5-9c82-0766a2a679e9device_caa0fc07-0456-4e86-ac52-e18524843fa8device_93d0045d-39ba-48e6-b704-3599b39f59f9=VmDevice:{id='VmDeviceId:{deviceId='93d0045d-39ba-48e6-b704-3599b39f59f9', vmId='1c7e966f-29e8-4788-b64c-86e52f415061'}', device='spicevmc', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=3}', managed='false', plugged='true', readOnly='false', deviceAlias='channel2', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}}, display=qxl, timeOffset=0, nice=0, maxMemSize=16384, maxMemSlots=16, bootMenuEnable=false, memSize=4096} 2017-10-01 16:34:55,294-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] FINISH, CreateVDSCommand, log id: 7fe35141 2017-10-01 16:34:55,297-04 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] FINISH, CreateVmVDSCommand, return: WaitForLaunch, log id: 31b98765 2017-10-01 16:34:55,298-04 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] Lock freed to object 'EngineLock:{exclusiveLocks='[1c7e966f-29e8-4788-b64c-86e52f415061=VM]', sharedLocks=''}' 2017-10-01 16:34:55,306-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] EVENT_ID: USER_STARTED_VM(153), Correlation ID: ef0a1d32-bc65-4a25-a600-995f2ad50edb, Job ID: 4ae3ca02-cd32-41f8-9eeb-4a836661152f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM udb was started by jas@eecs.yorku.ca-authz (Host: virt1). 2017-10-01 16:34:55,328-04 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] Running command: RunVmCommand internal: false. Entities affected : ID: d56363ad-9cb1-4a70-a33a-ce439b044d7d Type: VMAction group RUN_VM with role type USER 2017-10-01 16:34:55,449-04 INFO [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] START, UpdateVmDynamicDataVDSCommand( UpdateVmDynamicDataVDSCommandParameters:{runAsync='true', hostId='null', vmId='d56363ad-9cb1-4a70-a33a-ce439b044d7d', vmDynamic='org.ovirt.engine.core.common.businessentities.VmDynamic@f495f499'}), log id: e3cac2b 2017-10-01 16:34:55,451-04 INFO [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] FINISH, UpdateVmDynamicDataVDSCommand, log id: e3cac2b 2017-10-01 16:34:55,456-04 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] START, CreateVmVDSCommand( CreateVmVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vmId='d56363ad-9cb1-4a70-a33a-ce439b044d7d', vm='VM [svn]'}), log id: 13faf221 2017-10-01 16:34:55,458-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] START, CreateVDSCommand(HostName = virt3, CreateVmVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vmId='d56363ad-9cb1-4a70-a33a-ce439b044d7d', vm='VM [svn]'}), log id: 22539b83 2017-10-01 16:34:55,461-04 INFO [org.ovirt.engine.core.vdsbroker.builder.vminfo.VmInfoBuilderImpl] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] Bootable disk 'fcae8e52-ab63-4d21-8018-8a90ebb1f35a' set to index '0' 2017-10-01 16:34:55,477-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] VM {memGuaranteedSize=4096, smpThreadsPerCore=1, cpuType=SandyBridge, vmId=d56363ad-9cb1-4a70-a33a-ce439b044d7d, acpiEnable=true, vmType=kvm, smp=4, smpCoresPerSocket=1, emulatedMachine=pc-i440fx-rhel7.3.0, smartcardEnable=false, guestNumaNodes=[{memory=4096, cpus=0,1,2,3, nodeIndex=0}], transparentHugePages=true, displayNetwork=yorku, vmName=svn, maxVCpus=16, kvmEnable=true, devices=[{address={bus=0x00, domain=0x0000, function=0x0, slot=0x02, type=pci}, type=video, specParams={heads=1, vram=32768}, device=qxl, deviceId=17a773ff-d339-4628-9e8d-8bfe62550932}, {type=graphics, specParams={fileTransferEnable=true, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,ssmartcard,susbredir, copyPasteEnable=true}, device=spice, deviceId=0000002e-002e-002e-002e-00000000036d}, {iface=ide, shared=false, path=, address={bus=1, controller=0, unit=0, type=drive, target=0}, readonly=true, index=2, type=disk, specParams={path=}, device=cdrom, deviceId=90de1795-9555-479b-8a82-5bcc7efa17d3}, {discard=false, shared=false, address={bus=0x00, domain=0x0000, function=0x0, slot=0x07, type=pci}, imageID=fcae8e52-ab63-4d21-8018-8a90ebb1f35a, format=raw, index=0, optional=false, type=disk, deviceId=fcae8e52-ab63-4d21-8018-8a90ebb1f35a, domainID=c8653dba-63ec-4447-872c-0ffcc6af9c0f, propagateErrors=off, iface=virtio, readonly=false, bootOrder=1, poolID=aca61eac-ed8a-484b-bbee-3645148f6405, volumeID=d6357103-21d8-4547-a270-9f411cbc84bc, specParams={}, device=disk}, {discard=false, shared=false, address={bus=0x00, domain=0x0000, function=0x0, slot=0x08, type=pci}, imageID=bebf0874-fcc4-4acf-84da-0d6898a3900a, format=raw, optional=false, type=disk, deviceId=bebf0874-fcc4-4acf-84da-0d6898a3900a, domainID=c8653dba-63ec-4447-872c-0ffcc6af9c0f, propagateErrors=off, iface=virtio, readonly=false, poolID=aca61eac-ed8a-484b-bbee-3645148f6405, volumeID=8b9860ba-be20-407a-a272-a0bf82bde445, specParams={}, device=disk}, {filter=vdsm-no-mac-spoofing, nicModel=pv, address={bus=0x00, domain=0x0000, function=0x0, slot=0x03, type=pci}, type=interface, specParams={inbound={}, outbound={}}, device=bridge, linkActive=true, deviceId=6a766172-f4b1-4ed4-84ee-b74b65ff8d4c, macAddr=00:1a:4a:32:3d:11, network=yorku}, {filter=vdsm-no-mac-spoofing, nicModel=pv, address={bus=0x00, domain=0x0000, function=0x0, slot=0x04, type=pci}, type=interface, specParams={inbound={}, outbound={}}, device=bridge, linkActive=true, deviceId=117cf962-4ff9-4b09-9383-741ee1f43b54, macAddr=00:1a:4a:32:3d:12, network=mrpriv}, {address={bus=0x00, domain=0x0000, function=0x2, slot=0x01, type=pci}, index=0, model=piix3-uhci, type=controller, specParams={}, device=usb, deviceId=23cbb675-029c-4772-bc9f-a315edd41b21}, {address={bus=0x00, domain=0x0000, function=0x0, slot=0x05, type=pci}, index=0, model=virtio-scsi, type=controller, specParams={}, device=scsi, deviceId=8e4c754b-8ac6-4584-8fe1-4dd08406accb}, {address={bus=0x00, domain=0x0000, function=0x0, slot=0x06, type=pci}, type=controller, specParams={}, device=virtio-serial, deviceId=0f3830bc-9371-4620-bd33-d7170b74e33d}, {model=virtio, type=rng, specParams={source=urandom}, device=virtio, deviceId=0cb162a4-17c7-4e63-8ad2-00c28056a797}], custom={device_b4a86393-6dfb-4d30-9cb2-2174ee1b1ad3device_58b4d0dc-c3f5-4ce5-888d-d6215a4a982e=VmDevice:{id='VmDeviceId:{deviceId='58b4d0dc-c3f5-4ce5-888d-d6215a4a982e', vmId='d56363ad-9cb1-4a70-a33a-ce439b044d7d'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=2}', managed='false', plugged='true', readOnly='false', deviceAlias='channel1', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_b4a86393-6dfb-4d30-9cb2-2174ee1b1ad3device_58b4d0dc-c3f5-4ce5-888d-d6215a4a982edevice_0e9f02f4-7590-4b92-a564-6314086bc158=VmDevice:{id='VmDeviceId:{deviceId='0e9f02f4-7590-4b92-a564-6314086bc158', vmId='d56363ad-9cb1-4a70-a33a-ce439b044d7d'}', device='spicevmc', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=3}', managed='false', plugged='true', readOnly='false', deviceAlias='channel2', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_b4a86393-6dfb-4d30-9cb2-2174ee1b1ad3=VmDevice:{id='VmDeviceId:{deviceId='b4a86393-6dfb-4d30-9cb2-2174ee1b1ad3', vmId='d56363ad-9cb1-4a70-a33a-ce439b044d7d'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=1}', managed='false', plugged='true', readOnly='false', deviceAlias='channel0', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_b4a86393-6dfb-4d30-9cb2-2174ee1b1ad3device_58b4d0dc-c3f5-4ce5-888d-d6215a4a982edevice_0e9f02f4-7590-4b92-a564-6314086bc158device_956e0903-a772-41d8-bf74-42325bc04f6f=VmDevice:{id='VmDeviceId:{deviceId='956e0903-a772-41d8-bf74-42325bc04f6f', vmId='d56363ad-9cb1-4a70-a33a-ce439b044d7d'}', device='ide', type='CONTROLLER', bootOrder='0', specParams='[]', address='{slot=0x01, bus=0x00, domain=0x0000, type=pci, function=0x1}', managed='false', plugged='true', readOnly='false', deviceAlias='ide', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}}, display=qxl, timeOffset=0, nice=0, maxMemSize=16384, maxMemSlots=16, bootMenuEnable=false, memSize=4096} 2017-10-01 16:34:55,907-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] FINISH, CreateVDSCommand, log id: 22539b83 2017-10-01 16:34:55,910-04 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] FINISH, CreateVmVDSCommand, return: WaitForLaunch, log id: 13faf221 2017-10-01 16:34:55,912-04 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] Lock freed to object 'EngineLock:{exclusiveLocks='[d56363ad-9cb1-4a70-a33a-ce439b044d7d=VM]', sharedLocks=''}' 2017-10-01 16:34:55,924-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] EVENT_ID: USER_STARTED_VM(153), Correlation ID: ef0a1d32-bc65-4a25-a600-995f2ad50edb, Job ID: 2af1dd23-b7a9-48ac-b8d1-dedea8646c2d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM svn was started by jas@eecs.yorku.ca-authz (Host: virt3). 2017-10-01 16:34:55,939-04 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] Running command: RunVmCommand internal: false. Entities affected : ID: aa487207-7ff4-465a-9d9b-2a103d50dc77 Type: VMAction group RUN_VM with role type USER 2017-10-01 16:34:56,024-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmsStatisticsFetcher] (DefaultQuartzScheduler4) [e361571] Fetched 2 VMs from VDS '45f8b331-842e-48e7-9df8-56adddb93836' 2017-10-01 16:34:56,030-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVDSCommand] (DefaultQuartzScheduler4) [e361571] START, FullListVDSCommand(HostName = virt1, FullListVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vmIds='[1c7e966f-29e8-4788-b64c-86e52f415061]'}), log id: 61715ff0 2017-10-01 16:34:56,056-04 INFO [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] START, UpdateVmDynamicDataVDSCommand( UpdateVmDynamicDataVDSCommandParameters:{runAsync='true', hostId='null', vmId='aa487207-7ff4-465a-9d9b-2a103d50dc77', vmDynamic='org.ovirt.engine.core.common.businessentities.VmDynamic@9e1fe61b'}), log id: 4153df26 2017-10-01 16:34:56,058-04 INFO [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] FINISH, UpdateVmDynamicDataVDSCommand, log id: 4153df26 2017-10-01 16:34:56,062-04 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] START, CreateVmVDSCommand( CreateVmVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vmId='aa487207-7ff4-465a-9d9b-2a103d50dc77', vm='VM [ftp1]'}), log id: 64bba238 2017-10-01 16:34:56,064-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] START, CreateVDSCommand(HostName = virt2, CreateVmVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vmId='aa487207-7ff4-465a-9d9b-2a103d50dc77', vm='VM [ftp1]'}), log id: 4dd83f9f 2017-10-01 16:34:56,068-04 INFO [org.ovirt.engine.core.vdsbroker.builder.vminfo.VmInfoBuilderImpl] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] Bootable disk '2052185c-bea2-48d5-b937-cf41e120610d' set to index '0' 2017-10-01 16:34:56,083-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] VM {memGuaranteedSize=1365, smpThreadsPerCore=1, cpuType=SandyBridge, vmId=aa487207-7ff4-465a-9d9b-2a103d50dc77, acpiEnable=true, vmType=kvm, smp=1, smpCoresPerSocket=1, emulatedMachine=pc-i440fx-rhel7.3.0, smartcardEnable=false, guestNumaNodes=[{memory=2048, cpus=0, nodeIndex=0}], transparentHugePages=true, displayNetwork=yorku, vmName=ftp1, maxVCpus=16, kvmEnable=true, devices=[{address={bus=0x00, domain=0x0000, function=0x0, slot=0x02, type=pci}, type=video, specParams={heads=1, vram=32768}, device=qxl, deviceId=ffb735a6-e983-45f0-9bdf-b68300e8e50d}, {type=graphics, specParams={fileTransferEnable=true, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,ssmartcard,susbredir, copyPasteEnable=true}, device=spice, deviceId=00000024-0024-0024-0024-0000000001c3}, {iface=ide, shared=false, path=, address={bus=1, controller=0, unit=0, type=drive, target=0}, readonly=true, index=2, type=disk, specParams={path=}, device=cdrom, deviceId=866d8670-a8ef-45a5-90c5-c1297d14c787}, {discard=false, shared=false, address={bus=0x00, domain=0x0000, function=0x0, slot=0x07, type=pci}, imageID=2052185c-bea2-48d5-b937-cf41e120610d, format=raw, index=0, optional=false, type=disk, deviceId=2052185c-bea2-48d5-b937-cf41e120610d, domainID=c8653dba-63ec-4447-872c-0ffcc6af9c0f, propagateErrors=off, iface=virtio, readonly=false, bootOrder=1, poolID=aca61eac-ed8a-484b-bbee-3645148f6405, volumeID=00eae63d-fee6-4e8a-81bd-81463dd86f3e, specParams={}, device=disk}, {filter=vdsm-no-mac-spoofing, nicModel=pv, address={bus=0x00, domain=0x0000, function=0x0, slot=0x03, type=pci}, bootOrder=2, type=interface, specParams={inbound={}, outbound={}}, device=bridge, linkActive=true, deviceId=59b50e7d-2abf-4e5b-82d1-39fa8278c8c1, macAddr=00:1a:4a:32:3d:0b, network=yorku}, {filter=vdsm-no-mac-spoofing, nicModel=pv, address={bus=0x00, domain=0x0000, function=0x0, slot=0x04, type=pci}, bootOrder=3, type=interface, specParams={inbound={}, outbound={}}, device=bridge, linkActive=true, deviceId=91e53092-f65e-4e8f-b33f-ca8ae11dd2e3, macAddr=00:1a:4a:32:3d:0c, network=mrpriv}, {address={bus=0x00, domain=0x0000, function=0x2, slot=0x01, type=pci}, index=0, model=piix3-uhci, type=controller, specParams={}, device=usb, deviceId=c11b7e9e-6f39-49fb-b56d-001e9e58b5db}, {address={bus=0x00, function=0x0, domain=0x0000, slot=0x08, type=pci}, type=balloon, specParams={model=virtio}, device=memballoon, deviceId=ac710c6b-e20e-4a02-be4a-355825d1091d}, {address={bus=0x00, domain=0x0000, function=0x0, slot=0x05, type=pci}, index=0, model=virtio-scsi, type=controller, specParams={}, device=scsi, deviceId=50dcba5e-ccb9-4ed1-9c99-bb217f77ee27}, {address={bus=0x00, domain=0x0000, function=0x0, slot=0x06, type=pci}, type=controller, specParams={}, device=virtio-serial, deviceId=efd4ab33-8bcb-40a5-bb55-1984a5cd565b}, {model=virtio, type=rng, specParams={source=urandom}, device=virtio, deviceId=adb87698-9ad2-4b1a-8adb-b84986b725c6}], custom={device_21a77b7d-9169-4b85-a9da-862116b3348bdevice_ad098866-fd36-4876-850e-3530893b2d8edevice_b03fbbe1-3200-4876-a047-a33ff6e5c28f=VmDevice:{id='VmDeviceId:{deviceId='b03fbbe1-3200-4876-a047-a33ff6e5c28f', vmId='aa487207-7ff4-465a-9d9b-2a103d50dc77'}', device='spicevmc', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=3}', managed='false', plugged='true', readOnly='false', deviceAlias='channel2', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_21a77b7d-9169-4b85-a9da-862116b3348b=VmDevice:{id='VmDeviceId:{deviceId='21a77b7d-9169-4b85-a9da-862116b3348b', vmId='aa487207-7ff4-465a-9d9b-2a103d50dc77'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=1}', managed='false', plugged='true', readOnly='false', deviceAlias='channel0', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_21a77b7d-9169-4b85-a9da-862116b3348bdevice_ad098866-fd36-4876-850e-3530893b2d8edevice_b03fbbe1-3200-4876-a047-a33ff6e5c28fdevice_d00873ad-f4ea-4dec-aa73-99aa3b82a5ad=VmDevice:{id='VmDeviceId:{deviceId='d00873ad-f4ea-4dec-aa73-99aa3b82a5ad', vmId='aa487207-7ff4-465a-9d9b-2a103d50dc77'}', device='ide', type='CONTROLLER', bootOrder='0', specParams='[]', address='{slot=0x01, bus=0x00, domain=0x0000, type=pci, function=0x1}', managed='false', plugged='true', readOnly='false', deviceAlias='ide', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_21a77b7d-9169-4b85-a9da-862116b3348bdevice_ad098866-fd36-4876-850e-3530893b2d8e=VmDevice:{id='VmDeviceId:{deviceId='ad098866-fd36-4876-850e-3530893b2d8e', vmId='aa487207-7ff4-465a-9d9b-2a103d50dc77'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=2}', managed='false', plugged='true', readOnly='false', deviceAlias='channel1', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}}, display=qxl, timeOffset=0, nice=0, maxMemSize=8192, maxMemSlots=16, bootMenuEnable=false, memSize=2048} 2017-10-01 16:34:56,084-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVDSCommand] (DefaultQuartzScheduler4) [e361571] FINISH, FullListVDSCommand, return: [{acpiEnable=true, emulatedMachine=pc-i440fx-rhel7.3.0, vmId=1c7e966f-29e8-4788-b64c-86e52f415061, guestDiskMapping={}, transparentHugePages=true, timeOffset=0, cpuType=SandyBridge, smp=2, guestNumaNodes=[Ljava.lang.Object;@5c30f6b6, smartcardEnable=false, custom={device_56041595-a1a1-4da5-9c82-0766a2a679e9device_caa0fc07-0456-4e86-ac52-e18524843fa8device_93d0045d-39ba-48e6-b704-3599b39f59f9device_5a114c4b-2119-4016-9a29-69f19deb09f1=VmDevice:{id='VmDeviceId:{deviceId='5a114c4b-2119-4016-9a29-69f19deb09f1', vmId='1c7e966f-29e8-4788-b64c-86e52f415061'}', device='ide', type='CONTROLLER', bootOrder='0', specParams='[]', address='{slot=0x01, bus=0x00, domain=0x0000, type=pci, function=0x1}', managed='false', plugged='true', readOnly='false', deviceAlias='ide', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_56041595-a1a1-4da5-9c82-0766a2a679e9device_caa0fc07-0456-4e86-ac52-e18524843fa8device_93d0045d-39ba-48e6-b704-3599b39f59f9=VmDevice:{id='VmDeviceId:{deviceId='93d0045d-39ba-48e6-b704-3599b39f59f9', vmId='1c7e966f-29e8-4788-b64c-86e52f415061'}', device='spicevmc', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=3}', managed='false', plugged='true', readOnly='false', deviceAlias='channel2', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_56041595-a1a1-4da5-9c82-0766a2a679e9=VmDevice:{id='VmDeviceId:{deviceId='56041595-a1a1-4da5-9c82-0766a2a679e9', vmId='1c7e966f-29e8-4788-b64c-86e52f415061'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=1}', managed='false', plugged='true', readOnly='false', deviceAlias='channel0', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_56041595-a1a1-4da5-9c82-0766a2a679e9device_caa0fc07-0456-4e86-ac52-e18524843fa8=VmDevice:{id='VmDeviceId:{deviceId='caa0fc07-0456-4e86-ac52-e18524843fa8', vmId='1c7e966f-29e8-4788-b64c-86e52f415061'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=2}', managed='false', plugged='true', readOnly='false', deviceAlias='channel1', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}}, vmType=kvm, memSize=4096, smpCoresPerSocket=1, vmName=udb, nice=0, status=WaitForLaunch, maxMemSize=16384, bootMenuEnable=false, pid=0, smpThreadsPerCore=1, memGuaranteedSize=4096, kvmEnable=true, displayNetwork=yorku, devices=[Ljava.lang.Object;@43e02951, display=qxl, maxVCpus=16, clientIp=, statusTime=4295529230, maxMemSlots=16}], log id: 61715ff0 2017-10-01 16:34:56,086-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler4) [e361571] Received a memballoon Device without an address when processing VM 1c7e966f-29e8-4788-b64c-86e52f415061 devices, skipping device: {device=memballoon, specParams={model=none}, type=balloon} 2017-10-01 16:34:56,086-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler4) [e361571] Received a virtio Device without an address when processing VM 1c7e966f-29e8-4788-b64c-86e52f415061 devices, skipping device: {device=virtio, specParams={source=urandom}, model=virtio, type=rng, deviceId=51dc3ead-f8e1-4c8e-b923-9f1b4b7fdbbe} 2017-10-01 16:34:56,086-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler4) [e361571] Received a spice Device without an address when processing VM 1c7e966f-29e8-4788-b64c-86e52f415061 devices, skipping device: {device=spice, specParams={fileTransferEnable=true, displayNetwork=yorku, copyPasteEnable=true, displayIp=130.63.94.34, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,ssmartcard,susbredir}, type=graphics, deviceId=05aeb875-80f2-4a28-9070-4975d8dc2b9b} 2017-10-01 16:34:56,346-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] FINISH, CreateVDSCommand, log id: 4dd83f9f 2017-10-01 16:34:56,348-04 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] FINISH, CreateVmVDSCommand, return: WaitForLaunch, log id: 64bba238 2017-10-01 16:34:56,350-04 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] Lock freed to object 'EngineLock:{exclusiveLocks='[aa487207-7ff4-465a-9d9b-2a103d50dc77=VM]', sharedLocks=''}' 2017-10-01 16:34:56,358-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] EVENT_ID: USER_STARTED_VM(153), Correlation ID: ef0a1d32-bc65-4a25-a600-995f2ad50edb, Job ID: a50283b9-72f4-413d-8e17-c195edd9dd37, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM ftp1 was started by jas@eecs.yorku.ca-authz (Host: virt2). 2017-10-01 16:34:56,372-04 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] Running command: RunVmCommand internal: false. Entities affected : ID: bfe9a524-2238-4bab-a91f-5dc6ed270896 Type: VMAction group RUN_VM with role type USER 2017-10-01 16:34:56,467-04 INFO [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] START, UpdateVmDynamicDataVDSCommand( UpdateVmDynamicDataVDSCommandParameters:{runAsync='true', hostId='null', vmId='bfe9a524-2238-4bab-a91f-5dc6ed270896', vmDynamic='org.ovirt.engine.core.common.businessentities.VmDynamic@8db1c2f5'}), log id: 29e9cdb 2017-10-01 16:34:56,469-04 INFO [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] FINISH, UpdateVmDynamicDataVDSCommand, log id: 29e9cdb 2017-10-01 16:34:56,471-04 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] START, CreateVmVDSCommand( CreateVmVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vmId='bfe9a524-2238-4bab-a91f-5dc6ed270896', vm='VM [pcmode]'}), log id: e12f43c 2017-10-01 16:34:56,472-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] START, CreateVDSCommand(HostName = virt4, CreateVmVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vmId='bfe9a524-2238-4bab-a91f-5dc6ed270896', vm='VM [pcmode]'}), log id: 49432ce5 2017-10-01 16:34:56,493-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] VM {memGuaranteedSize=4096, smpThreadsPerCore=1, cpuType=SandyBridge, vmId=bfe9a524-2238-4bab-a91f-5dc6ed270896, acpiEnable=true, vmType=kvm, smp=1, smpCoresPerSocket=1, emulatedMachine=pc-i440fx-rhel7.3.0, smartcardEnable=false, guestNumaNodes=[{memory=4096, cpus=0, nodeIndex=0}], transparentHugePages=true, displayNetwork=yorku, vmName=pcmode, maxVCpus=16, kvmEnable=true, devices=[{address={bus=0x00, domain=0x0000, function=0x0, slot=0x02, type=pci}, type=video, specParams={vgamem=16384, heads=1, vram=8192, ram=65536}, device=qxl, deviceId=4fca7d95-5390-4b71-a9c2-d69e76e7a4d9}, {type=graphics, specParams={fileTransferEnable=true, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,ssmartcard,susbredir, copyPasteEnable=true}, device=spice, deviceId=ca0f5b5f-d51f-4838-8042-a1e14722e179}, {iface=ide, shared=false, path=, address={bus=1, controller=0, unit=0, type=drive, target=0}, readonly=true, index=2, type=disk, specParams={path=}, device=cdrom, deviceId=9694a771-c419-4621-8083-d9ec838f125f}, {discard=false, shared=false, address={bus=0x00, domain=0x0000, function=0x0, slot=0x07, type=pci}, imageID=8377ab9a-ba36-48f3-9dcc-72f0e86bfe34, format=raw, optional=false, type=disk, deviceId=8377ab9a-ba36-48f3-9dcc-72f0e86bfe34, domainID=c8653dba-63ec-4447-872c-0ffcc6af9c0f, propagateErrors=off, iface=virtio, readonly=false, poolID=aca61eac-ed8a-484b-bbee-3645148f6405, volumeID=39fef4aa-bd72-4cbd-979e-c72339fab1fb, specParams={}, device=disk}, {filter=vdsm-no-mac-spoofing, nicModel=pv, address={bus=0x00, domain=0x0000, function=0x0, slot=0x03, type=pci}, type=interface, specParams={inbound={}, outbound={}}, device=bridge, linkActive=true, deviceId=8f8d600a-7b36-420d-a552-e2dd17b8b6f6, macAddr=00:1a:4a:32:3d:36, network=yorku}, {filter=vdsm-no-mac-spoofing, nicModel=pv, address={bus=0x00, domain=0x0000, function=0x0, slot=0x04, type=pci}, type=interface, specParams={inbound={}, outbound={}}, device=bridge, linkActive=true, deviceId=c561a74e-5cee-45bb-a6d6-5ab1570cd16a, macAddr=00:1a:4a:32:3d:37, network=mrpriv}, {address={bus=0x00, domain=0x0000, function=0x2, slot=0x01, type=pci}, index=0, model=piix3-uhci, type=controller, specParams={}, device=usb, deviceId=83140a41-462c-4101-8d5f-f0f5a76aa5c8}, {address={bus=0x00, domain=0x0000, function=0x0, slot=0x05, type=pci}, index=0, model=virtio-scsi, type=controller, specParams={}, device=scsi, deviceId=da5f2155-a60e-471a-9357-d2085d8c5b20}, {address={bus=0x00, domain=0x0000, function=0x0, slot=0x06, type=pci}, type=controller, specParams={}, device=virtio-serial, deviceId=deb8947e-b015-44db-9fbc-705e8c486eca}, {model=virtio, type=rng, specParams={source=urandom}, device=virtio, deviceId=8c62f470-4e04-4873-8c24-9076ad4b4f59}], custom={device_824b2add-bc03-46e1-9c2e-b9a6db1446ff=VmDevice:{id='VmDeviceId:{deviceId='824b2add-bc03-46e1-9c2e-b9a6db1446ff', vmId='bfe9a524-2238-4bab-a91f-5dc6ed270896'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=1}', managed='false', plugged='true', readOnly='false', deviceAlias='channel0', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_824b2add-bc03-46e1-9c2e-b9a6db1446ffdevice_aeaf7e01-9c68-4e9b-885f-bff2a751bd2e=VmDevice:{id='VmDeviceId:{deviceId='aeaf7e01-9c68-4e9b-885f-bff2a751bd2e', vmId='bfe9a524-2238-4bab-a91f-5dc6ed270896'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=2}', managed='false', plugged='true', readOnly='false', deviceAlias='channel1', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_824b2add-bc03-46e1-9c2e-b9a6db1446ffdevice_aeaf7e01-9c68-4e9b-885f-bff2a751bd2edevice_99751d4e-6998-4ece-ba2d-3938d4a037d6device_536ebb04-ccad-43a0-8df5-c58ff732ef52=VmDevice:{id='VmDeviceId:{deviceId='536ebb04-ccad-43a0-8df5-c58ff732ef52', vmId='bfe9a524-2238-4bab-a91f-5dc6ed270896'}', device='ide', type='CONTROLLER', bootOrder='0', specParams='[]', address='{slot=0x01, bus=0x00, domain=0x0000, type=pci, function=0x1}', managed='false', plugged='true', readOnly='false', deviceAlias='ide', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_824b2add-bc03-46e1-9c2e-b9a6db1446ffdevice_aeaf7e01-9c68-4e9b-885f-bff2a751bd2edevice_99751d4e-6998-4ece-ba2d-3938d4a037d6=VmDevice:{id='VmDeviceId:{deviceId='99751d4e-6998-4ece-ba2d-3938d4a037d6', vmId='bfe9a524-2238-4bab-a91f-5dc6ed270896'}', device='spicevmc', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=3}', managed='false', plugged='true', readOnly='false', deviceAlias='channel2', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}}, display=qxl, timeOffset=0, nice=0, maxMemSize=16384, maxMemSlots=16, bootMenuEnable=false, memSize=4096} 2017-10-01 16:34:56,964-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler7) [1fbaeae4] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 16:34:57,506-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] FINISH, CreateVDSCommand, log id: 49432ce5 2017-10-01 16:34:57,508-04 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] FINISH, CreateVmVDSCommand, return: WaitForLaunch, log id: e12f43c 2017-10-01 16:34:57,510-04 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] Lock freed to object 'EngineLock:{exclusiveLocks='[bfe9a524-2238-4bab-a91f-5dc6ed270896=VM]', sharedLocks=''}' 2017-10-01 16:34:57,514-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] EVENT_ID: USER_STARTED_VM(153), Correlation ID: ef0a1d32-bc65-4a25-a600-995f2ad50edb, Job ID: 02e372b0-3f8b-4ec0-ba35-4698dc90ed21, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM pcmode was started by jas@eecs.yorku.ca-authz (Host: virt4). 2017-10-01 16:34:57,524-04 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] Running command: RunVmCommand internal: false. Entities affected : ID: a82b4cc3-6ef6-4a8e-97ef-1d4ea1b7e07c Type: VMAction group RUN_VM with role type USER 2017-10-01 16:34:57,620-04 INFO [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] START, UpdateVmDynamicDataVDSCommand( UpdateVmDynamicDataVDSCommandParameters:{runAsync='true', hostId='null', vmId='a82b4cc3-6ef6-4a8e-97ef-1d4ea1b7e07c', vmDynamic='org.ovirt.engine.core.common.businessentities.VmDynamic@2ea2b80a'}), log id: 2c09a04 2017-10-01 16:34:57,621-04 INFO [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] FINISH, UpdateVmDynamicDataVDSCommand, log id: 2c09a04 2017-10-01 16:34:57,623-04 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] START, CreateVmVDSCommand( CreateVmVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vmId='a82b4cc3-6ef6-4a8e-97ef-1d4ea1b7e07c', vm='VM [indigo]'}), log id: 2ac933e3 2017-10-01 16:34:57,625-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] START, CreateVDSCommand(HostName = virt2, CreateVmVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vmId='a82b4cc3-6ef6-4a8e-97ef-1d4ea1b7e07c', vm='VM [indigo]'}), log id: 4280da37 2017-10-01 16:34:57,629-04 INFO [org.ovirt.engine.core.vdsbroker.builder.vminfo.VmInfoBuilderImpl] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] Bootable disk 'd4677fa0-279b-48c8-86d8-142214f2d14a' set to index '0' 2017-10-01 16:34:57,643-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] VM {memGuaranteedSize=16384, smpThreadsPerCore=1, cpuType=SandyBridge, vmId=a82b4cc3-6ef6-4a8e-97ef-1d4ea1b7e07c, acpiEnable=true, vmType=kvm, smp=4, smpCoresPerSocket=1, emulatedMachine=pc-i440fx-rhel7.3.0, smartcardEnable=false, guestNumaNodes=[{memory=16384, cpus=0,1,2,3, nodeIndex=0}], transparentHugePages=true, displayNetwork=yorku, vmName=indigo, maxVCpus=16, kvmEnable=true, devices=[{address={bus=0x00, domain=0x0000, function=0x0, slot=0x02, type=pci}, type=video, specParams={heads=1, vram=32768}, device=qxl, deviceId=6b2458bd-8115-48cb-a530-66432482f481}, {type=graphics, specParams={fileTransferEnable=true, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,ssmartcard,susbredir, copyPasteEnable=true}, device=spice, deviceId=0000002a-002a-002a-002a-000000000166}, {iface=ide, shared=false, path=, address={bus=1, controller=0, unit=0, type=drive, target=0}, readonly=true, index=2, type=disk, specParams={path=}, device=cdrom, deviceId=24e5b44a-1b63-4505-aebf-590b5ae0df95}, {discard=false, shared=false, address={bus=0, controller=0, unit=0, type=drive, target=0}, imageID=d4677fa0-279b-48c8-86d8-142214f2d14a, format=raw, index=0, optional=false, type=disk, deviceId=d4677fa0-279b-48c8-86d8-142214f2d14a, domainID=c8653dba-63ec-4447-872c-0ffcc6af9c0f, propagateErrors=off, iface=scsi, readonly=false, bootOrder=1, poolID=aca61eac-ed8a-484b-bbee-3645148f6405, volumeID=234e515e-6f66-4b94-a5f3-4485af855415, specParams={}, device=disk}, {filter=vdsm-no-mac-spoofing, nicModel=pv, address={bus=0x00, domain=0x0000, function=0x0, slot=0x03, type=pci}, type=interface, specParams={inbound={}, outbound={}}, device=bridge, linkActive=true, deviceId=a5826ace-ddd2-4c42-a9db-8d8e929c64eb, macAddr=00:1a:4a:32:3d:00, network=yorku}, {filter=vdsm-no-mac-spoofing, nicModel=pv, address={bus=0x00, domain=0x0000, function=0x0, slot=0x04, type=pci}, type=interface, specParams={inbound={}, outbound={}}, device=bridge, linkActive=true, deviceId=275d0257-c4e1-4efa-b3b4-d64511b111d4, macAddr=00:1a:4a:32:3d:01, network=mrpriv}, {address={bus=0x00, domain=0x0000, function=0x2, slot=0x01, type=pci}, index=0, model=piix3-uhci, type=controller, specParams={}, device=usb, deviceId=21db887b-244b-447a-95a9-a4c183e8505f}, {address={bus=0x00, domain=0x0000, function=0x0, slot=0x05, type=pci}, index=0, model=virtio-scsi, type=controller, specParams={}, device=scsi, deviceId=ceb7815d-b7b8-43f2-a1d0-b42cb837c1a6}, {address={bus=0x00, domain=0x0000, function=0x0, slot=0x06, type=pci}, type=controller, specParams={}, device=virtio-serial, deviceId=1bf251b7-8af4-4c80-972c-f6cc3f229656}, {model=virtio, type=rng, specParams={source=urandom}, device=virtio, deviceId=bf04a726-d678-46ec-8169-a05b7bf87954}], custom={device_9c16a7ba-80f8-467e-8e1b-610ee15a6cbedevice_135814bd-ae90-432b-b5a5-9173e8b44ce6device_f7655e22-cfd1-4388-ba00-35b7c29d2213=VmDevice:{id='VmDeviceId:{deviceId='f7655e22-cfd1-4388-ba00-35b7c29d2213', vmId='a82b4cc3-6ef6-4a8e-97ef-1d4ea1b7e07c'}', device='spicevmc', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=3}', managed='false', plugged='true', readOnly='false', deviceAlias='channel2', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_9c16a7ba-80f8-467e-8e1b-610ee15a6cbedevice_135814bd-ae90-432b-b5a5-9173e8b44ce6=VmDevice:{id='VmDeviceId:{deviceId='135814bd-ae90-432b-b5a5-9173e8b44ce6', vmId='a82b4cc3-6ef6-4a8e-97ef-1d4ea1b7e07c'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=2}', managed='false', plugged='true', readOnly='false', deviceAlias='channel1', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_9c16a7ba-80f8-467e-8e1b-610ee15a6cbe=VmDevice:{id='VmDeviceId:{deviceId='9c16a7ba-80f8-467e-8e1b-610ee15a6cbe', vmId='a82b4cc3-6ef6-4a8e-97ef-1d4ea1b7e07c'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=1}', managed='false', plugged='true', readOnly='false', deviceAlias='channel0', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_9c16a7ba-80f8-467e-8e1b-610ee15a6cbedevice_135814bd-ae90-432b-b5a5-9173e8b44ce6device_f7655e22-cfd1-4388-ba00-35b7c29d2213device_bee4d6b1-a28d-4dd6-a6a3-3bab1d73a117=VmDevice:{id='VmDeviceId:{deviceId='bee4d6b1-a28d-4dd6-a6a3-3bab1d73a117', vmId='a82b4cc3-6ef6-4a8e-97ef-1d4ea1b7e07c'}', device='ide', type='CONTROLLER', bootOrder='0', specParams='[]', address='{slot=0x01, bus=0x00, domain=0x0000, type=pci, function=0x1}', managed='false', plugged='true', readOnly='false', deviceAlias='ide', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}}, display=qxl, timeOffset=0, nice=0, maxMemSize=65536, maxMemSlots=16, bootMenuEnable=false, memSize=16384} 2017-10-01 16:34:57,803-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] FINISH, CreateVDSCommand, log id: 4280da37 2017-10-01 16:34:57,806-04 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] FINISH, CreateVmVDSCommand, return: WaitForLaunch, log id: 2ac933e3 2017-10-01 16:34:57,807-04 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] Lock freed to object 'EngineLock:{exclusiveLocks='[a82b4cc3-6ef6-4a8e-97ef-1d4ea1b7e07c=VM]', sharedLocks=''}' 2017-10-01 16:34:57,811-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] EVENT_ID: USER_STARTED_VM(153), Correlation ID: ef0a1d32-bc65-4a25-a600-995f2ad50edb, Job ID: 84f92459-c068-4a8c-a6a2-8e2db2344adc, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM indigo was started by jas@eecs.yorku.ca-authz (Host: virt2). 2017-10-01 16:34:57,821-04 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] Running command: RunVmCommand internal: false. Entities affected : ID: 07331c78-6e09-41b4-9b29-edda32f46351 Type: VMAction group RUN_VM with role type USER 2017-10-01 16:34:57,914-04 INFO [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] START, UpdateVmDynamicDataVDSCommand( UpdateVmDynamicDataVDSCommandParameters:{runAsync='true', hostId='null', vmId='07331c78-6e09-41b4-9b29-edda32f46351', vmDynamic='org.ovirt.engine.core.common.businessentities.VmDynamic@a654dd'}), log id: 3f15a7c8 2017-10-01 16:34:57,916-04 INFO [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] FINISH, UpdateVmDynamicDataVDSCommand, log id: 3f15a7c8 2017-10-01 16:34:57,918-04 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] START, CreateVmVDSCommand( CreateVmVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vmId='07331c78-6e09-41b4-9b29-edda32f46351', vm='VM [samba]'}), log id: 687d1283 2017-10-01 16:34:57,920-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] START, CreateVDSCommand(HostName = virt3, CreateVmVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vmId='07331c78-6e09-41b4-9b29-edda32f46351', vm='VM [samba]'}), log id: 730856eb 2017-10-01 16:34:57,923-04 INFO [org.ovirt.engine.core.vdsbroker.builder.vminfo.VmInfoBuilderImpl] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] Bootable disk 'd7340ce3-9f53-46cd-bd5c-e3e85adbf2d3' set to index '0' 2017-10-01 16:34:57,935-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] VM {memGuaranteedSize=4096, smpThreadsPerCore=1, cpuType=SandyBridge, vmId=07331c78-6e09-41b4-9b29-edda32f46351, acpiEnable=true, vmType=kvm, smp=4, smpCoresPerSocket=1, emulatedMachine=pc-i440fx-rhel7.3.0, smartcardEnable=false, guestNumaNodes=[{memory=4096, cpus=0,1,2,3, nodeIndex=0}], transparentHugePages=true, displayNetwork=yorku, vmName=samba, maxVCpus=16, kvmEnable=true, devices=[{address={bus=0x00, domain=0x0000, function=0x0, slot=0x02, type=pci}, type=video, specParams={heads=1, vram=32768}, device=qxl, deviceId=1e91d9e4-cd7e-424a-b399-a059d722a4c7}, {type=graphics, specParams={fileTransferEnable=true, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,ssmartcard,susbredir, copyPasteEnable=true}, device=spice, deviceId=00000033-0033-0033-0033-00000000028f}, {iface=ide, shared=false, path=, address={bus=1, controller=0, unit=0, type=drive, target=0}, readonly=true, index=2, type=disk, specParams={path=}, device=cdrom, deviceId=ec528c24-4059-4e77-8696-78e662a2790a}, {discard=false, shared=false, address={bus=0x00, domain=0x0000, function=0x0, slot=0x07, type=pci}, imageID=d7340ce3-9f53-46cd-bd5c-e3e85adbf2d3, format=raw, index=0, optional=false, type=disk, deviceId=d7340ce3-9f53-46cd-bd5c-e3e85adbf2d3, domainID=c8653dba-63ec-4447-872c-0ffcc6af9c0f, propagateErrors=off, iface=virtio, readonly=false, bootOrder=1, poolID=aca61eac-ed8a-484b-bbee-3645148f6405, volumeID=ce4eac97-09bd-4f8b-b1e7-2c70ec2f7eff, specParams={}, device=disk}, {filter=vdsm-no-mac-spoofing, nicModel=pv, address={bus=0x00, domain=0x0000, function=0x0, slot=0x03, type=pci}, type=interface, specParams={inbound={}, outbound={}}, device=bridge, linkActive=true, deviceId=d605ffe2-a025-431e-9c3e-bdd4572a0cd3, macAddr=00:1a:4a:32:3d:1b, network=yorku}, {filter=vdsm-no-mac-spoofing, nicModel=pv, address={bus=0x00, domain=0x0000, function=0x0, slot=0x04, type=pci}, type=interface, specParams={inbound={}, outbound={}}, device=bridge, linkActive=true, deviceId=db6aefd8-48a1-4324-b876-b22894312449, macAddr=00:1a:4a:32:3d:1c, network=mrpriv}, {address={bus=0x00, domain=0x0000, function=0x2, slot=0x01, type=pci}, index=0, model=piix3-uhci, type=controller, specParams={}, device=usb, deviceId=99861476-c83c-40cc-b022-b9b4bb9d5158}, {address={bus=0x00, domain=0x0000, function=0x0, slot=0x05, type=pci}, index=0, model=virtio-scsi, type=controller, specParams={}, device=scsi, deviceId=a1b52344-a253-497b-811f-06df94e1c5b9}, {address={bus=0x00, domain=0x0000, function=0x0, slot=0x06, type=pci}, type=controller, specParams={}, device=virtio-serial, deviceId=22bbe896-242a-4334-b962-dafb119ab8e5}, {model=virtio, type=rng, specParams={source=urandom}, device=virtio, deviceId=5d5bdae6-cfbe-4138-a9c0-98b93d7cb32a}], custom={device_f9fabb45-65d0-44aa-8519-442f0490daabdevice_720907bd-38e0-4e2e-82b3-0c09e2d8c76edevice_4c7ee575-6e86-4fe7-91ed-3c9cf3a71bb5device_1ee85fb3-575f-4aa6-8664-16938022b9e2=VmDevice:{id='VmDeviceId:{deviceId='1ee85fb3-575f-4aa6-8664-16938022b9e2', vmId='07331c78-6e09-41b4-9b29-edda32f46351'}', device='ide', type='CONTROLLER', bootOrder='0', specParams='[]', address='{slot=0x01, bus=0x00, domain=0x0000, type=pci, function=0x1}', managed='false', plugged='true', readOnly='false', deviceAlias='ide', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_f9fabb45-65d0-44aa-8519-442f0490daab=VmDevice:{id='VmDeviceId:{deviceId='f9fabb45-65d0-44aa-8519-442f0490daab', vmId='07331c78-6e09-41b4-9b29-edda32f46351'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=1}', managed='false', plugged='true', readOnly='false', deviceAlias='channel0', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_f9fabb45-65d0-44aa-8519-442f0490daabdevice_720907bd-38e0-4e2e-82b3-0c09e2d8c76e=VmDevice:{id='VmDeviceId:{deviceId='720907bd-38e0-4e2e-82b3-0c09e2d8c76e', vmId='07331c78-6e09-41b4-9b29-edda32f46351'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=2}', managed='false', plugged='true', readOnly='false', deviceAlias='channel1', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_f9fabb45-65d0-44aa-8519-442f0490daabdevice_720907bd-38e0-4e2e-82b3-0c09e2d8c76edevice_4c7ee575-6e86-4fe7-91ed-3c9cf3a71bb5=VmDevice:{id='VmDeviceId:{deviceId='4c7ee575-6e86-4fe7-91ed-3c9cf3a71bb5', vmId='07331c78-6e09-41b4-9b29-edda32f46351'}', device='spicevmc', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=3}', managed='false', plugged='true', readOnly='false', deviceAlias='channel2', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}}, display=qxl, timeOffset=0, nice=0, maxMemSize=16384, maxMemSlots=16, bootMenuEnable=false, memSize=4096} 2017-10-01 16:34:57,949-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] FINISH, CreateVDSCommand, log id: 730856eb 2017-10-01 16:34:57,952-04 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] FINISH, CreateVmVDSCommand, return: WaitForLaunch, log id: 687d1283 2017-10-01 16:34:57,956-04 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] Lock freed to object 'EngineLock:{exclusiveLocks='[07331c78-6e09-41b4-9b29-edda32f46351=VM]', sharedLocks=''}' 2017-10-01 16:34:57,967-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] EVENT_ID: USER_STARTED_VM(153), Correlation ID: ef0a1d32-bc65-4a25-a600-995f2ad50edb, Job ID: 831af959-a88a-49dd-97a8-fbe3f8c74394, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM samba was started by jas@eecs.yorku.ca-authz (Host: virt3). 2017-10-01 16:34:57,981-04 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] Running command: RunVmCommand internal: false. Entities affected : ID: f87ec08d-0fdf-42ba-8aac-c66c404d7aae Type: VMAction group RUN_VM with role type USER 2017-10-01 16:34:58,069-04 INFO [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] START, UpdateVmDynamicDataVDSCommand( UpdateVmDynamicDataVDSCommandParameters:{runAsync='true', hostId='null', vmId='f87ec08d-0fdf-42ba-8aac-c66c404d7aae', vmDynamic='org.ovirt.engine.core.common.businessentities.VmDynamic@1a2bca7c'}), log id: f86d41d 2017-10-01 16:34:58,071-04 INFO [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] FINISH, UpdateVmDynamicDataVDSCommand, log id: f86d41d 2017-10-01 16:34:58,073-04 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] START, CreateVmVDSCommand( CreateVmVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vmId='f87ec08d-0fdf-42ba-8aac-c66c404d7aae', vm='VM [webapp]'}), log id: 7e7e46c2 2017-10-01 16:34:58,074-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] START, CreateVDSCommand(HostName = virt4, CreateVmVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vmId='f87ec08d-0fdf-42ba-8aac-c66c404d7aae', vm='VM [webapp]'}), log id: 5d01edb1 2017-10-01 16:34:58,078-04 INFO [org.ovirt.engine.core.vdsbroker.builder.vminfo.VmInfoBuilderImpl] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] Bootable disk 'cb40caad-cb8c-432f-8128-2281d31d58c1' set to index '0' 2017-10-01 16:34:58,092-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] VM {memGuaranteedSize=4096, smpThreadsPerCore=1, cpuType=SandyBridge, vmId=f87ec08d-0fdf-42ba-8aac-c66c404d7aae, acpiEnable=true, vmType=kvm, smp=4, smpCoresPerSocket=1, emulatedMachine=pc-i440fx-rhel7.3.0, smartcardEnable=false, guestNumaNodes=[{memory=4096, cpus=0,1,2,3, nodeIndex=0}], transparentHugePages=true, displayNetwork=yorku, vmName=webapp, maxVCpus=16, kvmEnable=true, devices=[{address={bus=0x00, domain=0x0000, function=0x0, slot=0x02, type=pci}, type=video, specParams={heads=1, vram=32768}, device=qxl, deviceId=75043493-c6a5-4f88-9ded-a943407104f8}, {type=graphics, specParams={fileTransferEnable=true, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,ssmartcard,susbredir, copyPasteEnable=true}, device=spice, deviceId=0000001e-001e-001e-001e-0000000002bc}, {iface=ide, shared=false, path=, address={bus=1, controller=0, unit=0, type=drive, target=0}, readonly=true, index=2, type=disk, specParams={path=}, device=cdrom, deviceId=7944aa6e-7ea9-4a6d-9038-d7a49fea48c6}, {discard=false, shared=false, address={bus=0x00, domain=0x0000, function=0x0, slot=0x07, type=pci}, imageID=cb40caad-cb8c-432f-8128-2281d31d58c1, format=raw, index=0, optional=false, type=disk, deviceId=cb40caad-cb8c-432f-8128-2281d31d58c1, domainID=c8653dba-63ec-4447-872c-0ffcc6af9c0f, propagateErrors=off, iface=virtio, readonly=false, bootOrder=1, poolID=aca61eac-ed8a-484b-bbee-3645148f6405, volumeID=c66b1f4f-1221-4d5e-816d-94ad2a6b38dc, specParams={}, device=disk}, {discard=false, shared=false, address={bus=0x00, domain=0x0000, function=0x0, slot=0x08, type=pci}, imageID=31158fa7-1b82-450c-b084-058b434dfbf4, format=raw, optional=false, type=disk, deviceId=31158fa7-1b82-450c-b084-058b434dfbf4, domainID=c8653dba-63ec-4447-872c-0ffcc6af9c0f, propagateErrors=off, iface=virtio, readonly=false, poolID=aca61eac-ed8a-484b-bbee-3645148f6405, volumeID=f9555868-a4db-4a3c-9a38-876c7c7902c9, specParams={}, device=disk}, {filter=vdsm-no-mac-spoofing, nicModel=pv, address={bus=0x00, domain=0x0000, function=0x0, slot=0x03, type=pci}, type=interface, specParams={inbound={}, outbound={}}, device=bridge, linkActive=true, deviceId=67a61b78-18e2-447b-bcc5-b7101da66bbf, macAddr=00:1a:4a:32:3d:0f, network=yorku}, {filter=vdsm-no-mac-spoofing, nicModel=pv, address={bus=0x00, domain=0x0000, function=0x0, slot=0x04, type=pci}, type=interface, specParams={inbound={}, outbound={}}, device=bridge, linkActive=true, deviceId=23169f62-2677-4b0c-8a0b-93dd9e0ecc4f, macAddr=00:1a:4a:32:3d:10, network=mrpriv}, {address={bus=0x00, domain=0x0000, function=0x2, slot=0x01, type=pci}, index=0, model=piix3-uhci, type=controller, specParams={}, device=usb, deviceId=a21f9054-7c8a-46de-b744-d97400effc3e}, {address={bus=0x00, domain=0x0000, function=0x0, slot=0x05, type=pci}, index=0, model=virtio-scsi, type=controller, specParams={}, device=scsi, deviceId=660e8c37-9367-4a7c-8e58-3ec54697c3a6}, {address={bus=0x00, domain=0x0000, function=0x0, slot=0x06, type=pci}, type=controller, specParams={}, device=virtio-serial, deviceId=5c5d3139-a4cf-4f04-8f6d-aa35bd31a3d8}, {model=virtio, type=rng, specParams={source=urandom}, device=virtio, deviceId=5503db77-c4b8-4c54-9644-5b479c7ae106}], custom={device_b626f5c1-2022-4b58-a584-0c3b5b743a6bdevice_3c2d8356-f53a-4d8b-b8f9-3702b2619903device_5e4f0277-ca58-4f3f-87ad-4c836454484e=VmDevice:{id='VmDeviceId:{deviceId='5e4f0277-ca58-4f3f-87ad-4c836454484e', vmId='f87ec08d-0fdf-42ba-8aac-c66c404d7aae'}', device='spicevmc', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=3}', managed='false', plugged='true', readOnly='false', deviceAlias='channel2', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_b626f5c1-2022-4b58-a584-0c3b5b743a6bdevice_3c2d8356-f53a-4d8b-b8f9-3702b2619903device_5e4f0277-ca58-4f3f-87ad-4c836454484edevice_cd06efc2-9ec3-4642-858c-fd2d309e830d=VmDevice:{id='VmDeviceId:{deviceId='cd06efc2-9ec3-4642-858c-fd2d309e830d', vmId='f87ec08d-0fdf-42ba-8aac-c66c404d7aae'}', device='ide', type='CONTROLLER', bootOrder='0', specParams='[]', address='{slot=0x01, bus=0x00, domain=0x0000, type=pci, function=0x1}', managed='false', plugged='true', readOnly='false', deviceAlias='ide', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_b626f5c1-2022-4b58-a584-0c3b5b743a6bdevice_3c2d8356-f53a-4d8b-b8f9-3702b2619903=VmDevice:{id='VmDeviceId:{deviceId='3c2d8356-f53a-4d8b-b8f9-3702b2619903', vmId='f87ec08d-0fdf-42ba-8aac-c66c404d7aae'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=2}', managed='false', plugged='true', readOnly='false', deviceAlias='channel1', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_b626f5c1-2022-4b58-a584-0c3b5b743a6b=VmDevice:{id='VmDeviceId:{deviceId='b626f5c1-2022-4b58-a584-0c3b5b743a6b', vmId='f87ec08d-0fdf-42ba-8aac-c66c404d7aae'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=1}', managed='false', plugged='true', readOnly='false', deviceAlias='channel0', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}}, display=qxl, timeOffset=0, nice=0, maxMemSize=16384, maxMemSlots=16, bootMenuEnable=false, memSize=4096} 2017-10-01 16:34:58,415-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] FINISH, CreateVDSCommand, log id: 5d01edb1 2017-10-01 16:34:58,417-04 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] FINISH, CreateVmVDSCommand, return: WaitForLaunch, log id: 7e7e46c2 2017-10-01 16:34:58,417-04 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] Lock freed to object 'EngineLock:{exclusiveLocks='[f87ec08d-0fdf-42ba-8aac-c66c404d7aae=VM]', sharedLocks=''}' 2017-10-01 16:34:58,421-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] EVENT_ID: USER_STARTED_VM(153), Correlation ID: ef0a1d32-bc65-4a25-a600-995f2ad50edb, Job ID: 12edd07c-ff48-4d5a-967b-cc767a09d706, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM webapp was started by jas@eecs.yorku.ca-authz (Host: virt4). 2017-10-01 16:34:58,431-04 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] Running command: RunVmCommand internal: false. Entities affected : ID: ae65a104-ed3f-40e0-ba92-5708cf338918 Type: VMAction group RUN_VM with role type USER 2017-10-01 16:34:58,518-04 INFO [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] START, UpdateVmDynamicDataVDSCommand( UpdateVmDynamicDataVDSCommandParameters:{runAsync='true', hostId='null', vmId='ae65a104-ed3f-40e0-ba92-5708cf338918', vmDynamic='org.ovirt.engine.core.common.businessentities.VmDynamic@33f29297'}), log id: 7a435da 2017-10-01 16:34:58,520-04 INFO [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] FINISH, UpdateVmDynamicDataVDSCommand, log id: 7a435da 2017-10-01 16:34:58,522-04 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] START, CreateVmVDSCommand( CreateVmVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vmId='ae65a104-ed3f-40e0-ba92-5708cf338918', vm='VM [vpn2]'}), log id: 2cb5e4 2017-10-01 16:34:58,525-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] START, CreateVDSCommand(HostName = virt1, CreateVmVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vmId='ae65a104-ed3f-40e0-ba92-5708cf338918', vm='VM [vpn2]'}), log id: 7315effc 2017-10-01 16:34:58,528-04 INFO [org.ovirt.engine.core.vdsbroker.builder.vminfo.VmInfoBuilderImpl] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] Bootable disk '3fb9f28b-05be-4cbc-b106-6f7122da6de0' set to index '0' 2017-10-01 16:34:58,548-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] VM {memGuaranteedSize=1365, smpThreadsPerCore=1, cpuType=SandyBridge, vmId=ae65a104-ed3f-40e0-ba92-5708cf338918, acpiEnable=true, vmType=kvm, smp=2, smpCoresPerSocket=1, emulatedMachine=pc-i440fx-rhel7.3.0, smartcardEnable=false, guestNumaNodes=[{memory=2048, cpus=0,1, nodeIndex=0}], transparentHugePages=true, displayNetwork=yorku, vmName=vpn2, maxVCpus=16, kvmEnable=true, devices=[{address={bus=0x00, domain=0x0000, function=0x0, slot=0x02, type=pci}, type=video, specParams={vgamem=16384, heads=1, ram=65536, vram=8192}, device=qxl, deviceId=86ba934b-084b-4fb4-8916-e6e075515944}, {type=graphics, specParams={fileTransferEnable=true, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,ssmartcard,susbredir, copyPasteEnable=true}, device=spice, deviceId=f4fc08c8-a53e-48ef-be8e-6d847f92d837}, {iface=ide, shared=false, path=, address={bus=1, controller=0, unit=0, type=drive, target=0}, readonly=true, index=2, type=disk, specParams={path=}, device=cdrom, deviceId=d94158f7-1b40-4660-9c7e-b75dd3d5d8e2}, {discard=false, shared=false, address={bus=0x00, domain=0x0000, function=0x0, slot=0x07, type=pci}, imageID=3fb9f28b-05be-4cbc-b106-6f7122da6de0, format=raw, index=0, optional=false, type=disk, deviceId=3fb9f28b-05be-4cbc-b106-6f7122da6de0, domainID=c8653dba-63ec-4447-872c-0ffcc6af9c0f, propagateErrors=off, iface=virtio, readonly=false, bootOrder=1, poolID=aca61eac-ed8a-484b-bbee-3645148f6405, volumeID=e987a5a4-1f3a-4a73-ae56-d9c2bdc28933, specParams={}, device=disk}, {filter=vdsm-no-mac-spoofing, nicModel=pv, address={bus=0x00, domain=0x0000, function=0x0, slot=0x03, type=pci}, type=interface, specParams={inbound={}, outbound={}}, device=bridge, linkActive=true, deviceId=d0baa757-f51e-4d84-ad04-972fe75a60f4, macAddr=00:1a:4a:32:3d:32, network=yorku}, {filter=vdsm-no-mac-spoofing, nicModel=pv, address={bus=0x00, domain=0x0000, function=0x0, slot=0x04, type=pci}, type=interface, specParams={inbound={}, outbound={}}, device=bridge, linkActive=true, deviceId=740f2df7-39a6-4b26-84d0-6b27967d0391, macAddr=00:1a:4a:32:3d:33, network=mrpriv}, {address={bus=0x00, domain=0x0000, function=0x2, slot=0x01, type=pci}, index=0, model=piix3-uhci, type=controller, specParams={}, device=usb, deviceId=d2ae2aca-a755-4f33-944c-72e9f44cc435}, {address={bus=0x00, function=0x0, domain=0x0000, slot=0x08, type=pci}, type=balloon, specParams={model=virtio}, device=memballoon, deviceId=48ebdca0-4308-4130-bca2-88838a332309}, {address={bus=0x00, domain=0x0000, function=0x0, slot=0x05, type=pci}, index=0, model=virtio-scsi, type=controller, specParams={}, device=scsi, deviceId=83e7da21-4139-4413-b76f-6746368088e9}, {address={bus=0x00, domain=0x0000, function=0x0, slot=0x06, type=pci}, type=controller, specParams={}, device=virtio-serial, deviceId=9d000a12-9942-4d87-bdb1-aa07589b7932}, {model=virtio, type=rng, specParams={source=urandom}, device=virtio, deviceId=768f5bab-9e12-4e46-a88b-7bdaee382c08}], custom={device_ad0e60de-f05d-4a7d-beaa-50c74ff6e3b2device_caff46c6-9bc4-4657-82fa-74ddb1f395f3device_efaadb25-9ff8-4570-932c-97d8e90363cddevice_fd011cb3-5b43-4aa2-80ae-ac519e630280=VmDevice:{id='VmDeviceId:{deviceId='fd011cb3-5b43-4aa2-80ae-ac519e630280', vmId='ae65a104-ed3f-40e0-ba92-5708cf338918'}', device='ide', type='CONTROLLER', bootOrder='0', specParams='[]', address='{slot=0x01, bus=0x00, domain=0x0000, type=pci, function=0x1}', managed='false', plugged='true', readOnly='false', deviceAlias='ide', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_ad0e60de-f05d-4a7d-beaa-50c74ff6e3b2device_caff46c6-9bc4-4657-82fa-74ddb1f395f3device_efaadb25-9ff8-4570-932c-97d8e90363cd=VmDevice:{id='VmDeviceId:{deviceId='efaadb25-9ff8-4570-932c-97d8e90363cd', vmId='ae65a104-ed3f-40e0-ba92-5708cf338918'}', device='spicevmc', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=3}', managed='false', plugged='true', readOnly='false', deviceAlias='channel2', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_ad0e60de-f05d-4a7d-beaa-50c74ff6e3b2device_caff46c6-9bc4-4657-82fa-74ddb1f395f3=VmDevice:{id='VmDeviceId:{deviceId='caff46c6-9bc4-4657-82fa-74ddb1f395f3', vmId='ae65a104-ed3f-40e0-ba92-5708cf338918'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=2}', managed='false', plugged='true', readOnly='false', deviceAlias='channel1', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_ad0e60de-f05d-4a7d-beaa-50c74ff6e3b2=VmDevice:{id='VmDeviceId:{deviceId='ad0e60de-f05d-4a7d-beaa-50c74ff6e3b2', vmId='ae65a104-ed3f-40e0-ba92-5708cf338918'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=1}', managed='false', plugged='true', readOnly='false', deviceAlias='channel0', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}}, display=qxl, timeOffset=0, nice=0, maxMemSize=8192, maxMemSlots=16, bootMenuEnable=false, memSize=2048} 2017-10-01 16:34:58,698-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] FINISH, CreateVDSCommand, log id: 7315effc 2017-10-01 16:34:58,700-04 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] FINISH, CreateVmVDSCommand, return: WaitForLaunch, log id: 2cb5e4 2017-10-01 16:34:58,700-04 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] Lock freed to object 'EngineLock:{exclusiveLocks='[ae65a104-ed3f-40e0-ba92-5708cf338918=VM]', sharedLocks=''}' 2017-10-01 16:34:58,704-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] EVENT_ID: USER_STARTED_VM(153), Correlation ID: ef0a1d32-bc65-4a25-a600-995f2ad50edb, Job ID: 5d172da9-e7cb-4174-ad56-d1106e33b18b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM vpn2 was started by jas@eecs.yorku.ca-authz (Host: virt1). 2017-10-01 16:34:58,714-04 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] Running command: RunVmCommand internal: false. Entities affected : ID: 1325cebf-834e-4571-a3e1-82beb0021ffe Type: VMAction group RUN_VM with role type USER 2017-10-01 16:34:58,745-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmsStatisticsFetcher] (DefaultQuartzScheduler8) [599e0036] Fetched 4 VMs from VDS 'e1e71e9d-a0cd-47db-96c6-f724f6cea736' 2017-10-01 16:34:58,752-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVDSCommand] (DefaultQuartzScheduler8) [599e0036] START, FullListVDSCommand(HostName = virt3, FullListVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vmIds='[d56363ad-9cb1-4a70-a33a-ce439b044d7d, 07331c78-6e09-41b4-9b29-edda32f46351]'}), log id: 798404c4 2017-10-01 16:34:58,772-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVDSCommand] (DefaultQuartzScheduler8) [599e0036] FINISH, FullListVDSCommand, return: [{acpiEnable=true, emulatedMachine=pc-i440fx-rhel7.3.0, vmId=d56363ad-9cb1-4a70-a33a-ce439b044d7d, guestDiskMapping={}, transparentHugePages=true, timeOffset=0, cpuType=SandyBridge, smp=4, guestNumaNodes=[Ljava.lang.Object;@53ea9ee2, smartcardEnable=false, custom={device_b4a86393-6dfb-4d30-9cb2-2174ee1b1ad3=VmDevice:{id='VmDeviceId:{deviceId='b4a86393-6dfb-4d30-9cb2-2174ee1b1ad3', vmId='d56363ad-9cb1-4a70-a33a-ce439b044d7d'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=1}', managed='false', plugged='true', readOnly='false', deviceAlias='channel0', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_b4a86393-6dfb-4d30-9cb2-2174ee1b1ad3device_58b4d0dc-c3f5-4ce5-888d-d6215a4a982e=VmDevice:{id='VmDeviceId:{deviceId='58b4d0dc-c3f5-4ce5-888d-d6215a4a982e', vmId='d56363ad-9cb1-4a70-a33a-ce439b044d7d'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=2}', managed='false', plugged='true', readOnly='false', deviceAlias='channel1', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_b4a86393-6dfb-4d30-9cb2-2174ee1b1ad3device_58b4d0dc-c3f5-4ce5-888d-d6215a4a982edevice_0e9f02f4-7590-4b92-a564-6314086bc158=VmDevice:{id='VmDeviceId:{deviceId='0e9f02f4-7590-4b92-a564-6314086bc158', vmId='d56363ad-9cb1-4a70-a33a-ce439b044d7d'}', device='spicevmc', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=3}', managed='false', plugged='true', readOnly='false', deviceAlias='channel2', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_b4a86393-6dfb-4d30-9cb2-2174ee1b1ad3device_58b4d0dc-c3f5-4ce5-888d-d6215a4a982edevice_0e9f02f4-7590-4b92-a564-6314086bc158device_956e0903-a772-41d8-bf74-42325bc04f6f=VmDevice:{id='VmDeviceId:{deviceId='956e0903-a772-41d8-bf74-42325bc04f6f', vmId='d56363ad-9cb1-4a70-a33a-ce439b044d7d'}', device='ide', type='CONTROLLER', bootOrder='0', specParams='[]', address='{slot=0x01, bus=0x00, domain=0x0000, type=pci, function=0x1}', managed='false', plugged='true', readOnly='false', deviceAlias='ide', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}}, vmType=kvm, memSize=4096, smpCoresPerSocket=1, vmName=svn, nice=0, status=WaitForLaunch, maxMemSize=16384, bootMenuEnable=false, pid=0, smpThreadsPerCore=1, memGuaranteedSize=4096, kvmEnable=true, displayNetwork=yorku, devices=[Ljava.lang.Object;@67f4f78e, display=qxl, maxVCpus=16, clientIp=, statusTime=5262715210, maxMemSlots=16}, {acpiEnable=true, emulatedMachine=pc-i440fx-rhel7.3.0, vmId=07331c78-6e09-41b4-9b29-edda32f46351, guestDiskMapping={}, transparentHugePages=true, timeOffset=0, cpuType=SandyBridge, smp=4, guestNumaNodes=[Ljava.lang.Object;@dc26a5c, smartcardEnable=false, custom={device_f9fabb45-65d0-44aa-8519-442f0490daabdevice_720907bd-38e0-4e2e-82b3-0c09e2d8c76e=VmDevice:{id='VmDeviceId:{deviceId='720907bd-38e0-4e2e-82b3-0c09e2d8c76e', vmId='07331c78-6e09-41b4-9b29-edda32f46351'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=2}', managed='false', plugged='true', readOnly='false', deviceAlias='channel1', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_f9fabb45-65d0-44aa-8519-442f0490daab=VmDevice:{id='VmDeviceId:{deviceId='f9fabb45-65d0-44aa-8519-442f0490daab', vmId='07331c78-6e09-41b4-9b29-edda32f46351'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=1}', managed='false', plugged='true', readOnly='false', deviceAlias='channel0', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_f9fabb45-65d0-44aa-8519-442f0490daabdevice_720907bd-38e0-4e2e-82b3-0c09e2d8c76edevice_4c7ee575-6e86-4fe7-91ed-3c9cf3a71bb5=VmDevice:{id='VmDeviceId:{deviceId='4c7ee575-6e86-4fe7-91ed-3c9cf3a71bb5', vmId='07331c78-6e09-41b4-9b29-edda32f46351'}', device='spicevmc', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=3}', managed='false', plugged='true', readOnly='false', deviceAlias='channel2', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_f9fabb45-65d0-44aa-8519-442f0490daabdevice_720907bd-38e0-4e2e-82b3-0c09e2d8c76edevice_4c7ee575-6e86-4fe7-91ed-3c9cf3a71bb5device_1ee85fb3-575f-4aa6-8664-16938022b9e2=VmDevice:{id='VmDeviceId:{deviceId='1ee85fb3-575f-4aa6-8664-16938022b9e2', vmId='07331c78-6e09-41b4-9b29-edda32f46351'}', device='ide', type='CONTROLLER', bootOrder='0', specParams='[]', address='{slot=0x01, bus=0x00, domain=0x0000, type=pci, function=0x1}', managed='false', plugged='true', readOnly='false', deviceAlias='ide', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}}, vmType=kvm, memSize=4096, smpCoresPerSocket=1, vmName=samba, nice=0, status=WaitForLaunch, maxMemSize=16384, bootMenuEnable=false, pid=0, smpThreadsPerCore=1, memGuaranteedSize=4096, kvmEnable=true, displayNetwork=yorku, devices=[Ljava.lang.Object;@54b9ed04, display=qxl, maxVCpus=16, clientIp=, statusTime=5262715210, maxMemSlots=16}], log id: 798404c4 2017-10-01 16:34:58,774-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler8) [599e0036] Received a memballoon Device without an address when processing VM d56363ad-9cb1-4a70-a33a-ce439b044d7d devices, skipping device: {device=memballoon, specParams={model=none}, type=balloon} 2017-10-01 16:34:58,774-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler8) [599e0036] Received a virtio Device without an address when processing VM d56363ad-9cb1-4a70-a33a-ce439b044d7d devices, skipping device: {device=virtio, specParams={source=urandom}, model=virtio, type=rng, deviceId=0cb162a4-17c7-4e63-8ad2-00c28056a797} 2017-10-01 16:34:58,774-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler8) [599e0036] Received a spice Device without an address when processing VM d56363ad-9cb1-4a70-a33a-ce439b044d7d devices, skipping device: {device=spice, specParams={fileTransferEnable=true, displayNetwork=yorku, copyPasteEnable=true, displayIp=130.63.94.36, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,ssmartcard,susbredir}, type=graphics, deviceId=0000002e-002e-002e-002e-00000000036d} 2017-10-01 16:34:58,775-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler8) [599e0036] Received a memballoon Device without an address when processing VM 07331c78-6e09-41b4-9b29-edda32f46351 devices, skipping device: {device=memballoon, specParams={model=none}, type=balloon} 2017-10-01 16:34:58,775-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler8) [599e0036] Received a virtio Device without an address when processing VM 07331c78-6e09-41b4-9b29-edda32f46351 devices, skipping device: {device=virtio, specParams={source=urandom}, model=virtio, type=rng, deviceId=5d5bdae6-cfbe-4138-a9c0-98b93d7cb32a} 2017-10-01 16:34:58,775-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler8) [599e0036] Received a spice Device without an address when processing VM 07331c78-6e09-41b4-9b29-edda32f46351 devices, skipping device: {device=spice, specParams={fileTransferEnable=true, displayNetwork=yorku, copyPasteEnable=true, displayIp=130.63.94.36, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,ssmartcard,susbredir}, type=graphics, deviceId=00000033-0033-0033-0033-00000000028f} 2017-10-01 16:34:58,798-04 INFO [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] START, UpdateVmDynamicDataVDSCommand( UpdateVmDynamicDataVDSCommandParameters:{runAsync='true', hostId='null', vmId='1325cebf-834e-4571-a3e1-82beb0021ffe', vmDynamic='org.ovirt.engine.core.common.businessentities.VmDynamic@f501c0f1'}), log id: 7564ccd5 2017-10-01 16:34:58,800-04 INFO [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] FINISH, UpdateVmDynamicDataVDSCommand, log id: 7564ccd5 2017-10-01 16:34:58,803-04 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] START, CreateVmVDSCommand( CreateVmVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vmId='1325cebf-834e-4571-a3e1-82beb0021ffe', vm='VM [dns1]'}), log id: 710181b2 2017-10-01 16:34:58,805-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] START, CreateVDSCommand(HostName = virt3, CreateVmVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vmId='1325cebf-834e-4571-a3e1-82beb0021ffe', vm='VM [dns1]'}), log id: 298ca9e0 2017-10-01 16:34:58,809-04 INFO [org.ovirt.engine.core.vdsbroker.builder.vminfo.VmInfoBuilderImpl] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] Bootable disk '4073affa-0e79-479c-96bb-feb8bfee6ae4' set to index '0' 2017-10-01 16:34:58,824-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] VM {memGuaranteedSize=1365, smpThreadsPerCore=1, cpuType=SandyBridge, vmId=1325cebf-834e-4571-a3e1-82beb0021ffe, acpiEnable=true, vmType=kvm, smp=1, smpCoresPerSocket=1, emulatedMachine=pc-i440fx-rhel7.3.0, smartcardEnable=false, guestNumaNodes=[{memory=2048, cpus=0, nodeIndex=0}], transparentHugePages=true, displayNetwork=yorku, vmName=dns1, maxVCpus=16, kvmEnable=true, devices=[{address={bus=0x00, domain=0x0000, function=0x0, slot=0x02, type=pci}, type=video, specParams={heads=1, vram=32768}, device=qxl, deviceId=49347a24-5950-44bc-8820-096517ee5794}, {type=graphics, specParams={fileTransferEnable=true, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,ssmartcard,susbredir, copyPasteEnable=true}, device=spice, deviceId=3f758753-b44e-4641-a51c-e0554c983f33}, {iface=ide, shared=false, path=, address={bus=1, controller=0, unit=0, type=drive, target=0}, readonly=true, index=2, type=disk, specParams={path=}, device=cdrom, deviceId=e225ffe8-9920-4a7c-beca-6c4c777eca6f}, {discard=false, shared=false, address={bus=0x00, domain=0x0000, function=0x0, slot=0x07, type=pci}, imageID=4073affa-0e79-479c-96bb-feb8bfee6ae4, format=raw, index=0, optional=false, type=disk, deviceId=4073affa-0e79-479c-96bb-feb8bfee6ae4, domainID=c8653dba-63ec-4447-872c-0ffcc6af9c0f, propagateErrors=off, iface=virtio, readonly=false, bootOrder=1, poolID=aca61eac-ed8a-484b-bbee-3645148f6405, volumeID=5741a9a6-38c9-4e25-b4dc-2240acf6769a, specParams={}, device=disk}, {filter=vdsm-no-mac-spoofing, nicModel=pv, address={bus=0x00, domain=0x0000, function=0x0, slot=0x03, type=pci}, type=interface, specParams={inbound={}, outbound={}}, device=bridge, linkActive=true, deviceId=4afb9c7c-457f-4d5e-82c2-f506f022813f, macAddr=00:1a:4a:32:3d:0d, network=yorku}, {filter=vdsm-no-mac-spoofing, nicModel=pv, address={bus=0x00, domain=0x0000, function=0x0, slot=0x04, type=pci}, type=interface, specParams={inbound={}, outbound={}}, device=bridge, linkActive=true, deviceId=054ae0b5-895a-43ff-94ff-bcc31d782235, macAddr=00:1a:4a:32:3d:0e, network=mrpriv}, {address={bus=0x00, domain=0x0000, function=0x2, slot=0x01, type=pci}, index=0, model=piix3-uhci, type=controller, specParams={}, device=usb, deviceId=35fd9b71-be18-40a1-b606-c786fd811e1f}, {address={bus=0x00, domain=0x0000, function=0x0, slot=0x05, type=pci}, index=0, model=virtio-scsi, type=controller, specParams={}, device=scsi, deviceId=8c7269c6-d3fb-43e3-b68f-4d88b700bffc}, {address={bus=0x00, domain=0x0000, function=0x0, slot=0x06, type=pci}, type=controller, specParams={}, device=virtio-serial, deviceId=68ceea01-f283-437a-b937-4c239f941e2d}, {model=virtio, type=rng, specParams={source=urandom}, device=virtio, deviceId=05016f30-aebe-4665-9e6f-3b7ee82e0013}], custom={device_e8518a41-75b4-4047-9698-18caf5fc06f0device_fc446988-d249-4f93-80f6-c15a4ae3b8d9device_fca5daed-4de2-4882-b244-d15786b5267ddevice_9690f494-21ec-440d-ad07-62002944e0a7=VmDevice:{id='VmDeviceId:{deviceId='9690f494-21ec-440d-ad07-62002944e0a7', vmId='1325cebf-834e-4571-a3e1-82beb0021ffe'}', device='ide', type='CONTROLLER', bootOrder='0', specParams='[]', address='{slot=0x01, bus=0x00, domain=0x0000, type=pci, function=0x1}', managed='false', plugged='true', readOnly='false', deviceAlias='ide', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_e8518a41-75b4-4047-9698-18caf5fc06f0device_fc446988-d249-4f93-80f6-c15a4ae3b8d9device_fca5daed-4de2-4882-b244-d15786b5267d=VmDevice:{id='VmDeviceId:{deviceId='fca5daed-4de2-4882-b244-d15786b5267d', vmId='1325cebf-834e-4571-a3e1-82beb0021ffe'}', device='spicevmc', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=3}', managed='false', plugged='true', readOnly='false', deviceAlias='channel2', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_e8518a41-75b4-4047-9698-18caf5fc06f0=VmDevice:{id='VmDeviceId:{deviceId='e8518a41-75b4-4047-9698-18caf5fc06f0', vmId='1325cebf-834e-4571-a3e1-82beb0021ffe'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=1}', managed='false', plugged='true', readOnly='false', deviceAlias='channel0', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_e8518a41-75b4-4047-9698-18caf5fc06f0device_fc446988-d249-4f93-80f6-c15a4ae3b8d9=VmDevice:{id='VmDeviceId:{deviceId='fc446988-d249-4f93-80f6-c15a4ae3b8d9', vmId='1325cebf-834e-4571-a3e1-82beb0021ffe'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=2}', managed='false', plugged='true', readOnly='false', deviceAlias='channel1', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}}, display=qxl, timeOffset=0, nice=0, maxMemSize=8192, maxMemSlots=16, bootMenuEnable=false, memSize=2048} 2017-10-01 16:34:59,838-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] FINISH, CreateVDSCommand, log id: 298ca9e0 2017-10-01 16:34:59,841-04 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] FINISH, CreateVmVDSCommand, return: WaitForLaunch, log id: 710181b2 2017-10-01 16:34:59,841-04 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] Lock freed to object 'EngineLock:{exclusiveLocks='[1325cebf-834e-4571-a3e1-82beb0021ffe=VM]', sharedLocks=''}' 2017-10-01 16:34:59,845-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] EVENT_ID: USER_STARTED_VM(153), Correlation ID: ef0a1d32-bc65-4a25-a600-995f2ad50edb, Job ID: 27310415-d33c-45ba-aee5-7dc84841b6a0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM dns1 was started by jas@eecs.yorku.ca-authz (Host: virt3). 2017-10-01 16:34:59,857-04 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] Running command: RunVmCommand internal: false. Entities affected : ID: 3e5a5728-8940-4fe7-b4af-11fb8e12f116 Type: VMAction group RUN_VM with role type USER 2017-10-01 16:34:59,941-04 INFO [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] START, UpdateVmDynamicDataVDSCommand( UpdateVmDynamicDataVDSCommandParameters:{runAsync='true', hostId='null', vmId='3e5a5728-8940-4fe7-b4af-11fb8e12f116', vmDynamic='org.ovirt.engine.core.common.businessentities.VmDynamic@b2dffa25'}), log id: 1c072946 2017-10-01 16:34:59,943-04 INFO [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] FINISH, UpdateVmDynamicDataVDSCommand, log id: 1c072946 2017-10-01 16:34:59,945-04 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] START, CreateVmVDSCommand( CreateVmVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vmId='3e5a5728-8940-4fe7-b4af-11fb8e12f116', vm='VM [www]'}), log id: 2f167898 2017-10-01 16:34:59,947-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] START, CreateVDSCommand(HostName = virt4, CreateVmVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vmId='3e5a5728-8940-4fe7-b4af-11fb8e12f116', vm='VM [www]'}), log id: 51058938 2017-10-01 16:34:59,950-04 INFO [org.ovirt.engine.core.vdsbroker.builder.vminfo.VmInfoBuilderImpl] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] Bootable disk 'a8ded791-e974-4fe4-affb-d9a712bed681' set to index '0' 2017-10-01 16:34:59,964-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] VM {memGuaranteedSize=4096, smpThreadsPerCore=1, cpuType=SandyBridge, vmId=3e5a5728-8940-4fe7-b4af-11fb8e12f116, acpiEnable=true, vmType=kvm, smp=4, smpCoresPerSocket=1, emulatedMachine=pc-i440fx-rhel7.3.0, smartcardEnable=false, guestNumaNodes=[{memory=4096, cpus=0,1,2,3, nodeIndex=0}], transparentHugePages=true, displayNetwork=yorku, vmName=www, maxVCpus=16, kvmEnable=true, devices=[{address={bus=0x00, domain=0x0000, function=0x0, slot=0x02, type=pci}, type=video, specParams={heads=1, vram=32768}, device=qxl, deviceId=63a76544-442f-4747-aca0-d5600e379808}, {type=graphics, specParams={fileTransferEnable=true, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,ssmartcard,susbredir, copyPasteEnable=true}, device=spice, deviceId=0000002c-002c-002c-002c-000000000240}, {iface=ide, shared=false, path=, address={bus=1, controller=0, unit=0, type=drive, target=0}, readonly=true, index=2, type=disk, specParams={path=}, device=cdrom, deviceId=09a21314-adb5-430a-8ec5-834faa467465}, {discard=false, shared=false, address={bus=0x00, domain=0x0000, function=0x0, slot=0x07, type=pci}, imageID=a8ded791-e974-4fe4-affb-d9a712bed681, format=raw, index=0, optional=false, type=disk, deviceId=a8ded791-e974-4fe4-affb-d9a712bed681, domainID=c8653dba-63ec-4447-872c-0ffcc6af9c0f, propagateErrors=off, iface=virtio, readonly=false, bootOrder=1, poolID=aca61eac-ed8a-484b-bbee-3645148f6405, volumeID=b4f2bf5e-e4ae-42e6-8b4d-9bb2dcbabf96, specParams={}, device=disk}, {discard=false, shared=false, address={bus=0x00, domain=0x0000, function=0x0, slot=0x08, type=pci}, imageID=6d422988-fe44-4e45-aee0-5724b53c6bda, format=raw, optional=false, type=disk, deviceId=6d422988-fe44-4e45-aee0-5724b53c6bda, domainID=c8653dba-63ec-4447-872c-0ffcc6af9c0f, propagateErrors=off, iface=virtio, readonly=false, poolID=aca61eac-ed8a-484b-bbee-3645148f6405, volumeID=2df9ae7d-d3ad-470e-98d6-aeaa95091eaf, specParams={}, device=disk}, {filter=vdsm-no-mac-spoofing, nicModel=pv, address={bus=0x00, domain=0x0000, function=0x0, slot=0x03, type=pci}, type=interface, specParams={inbound={}, outbound={}}, device=bridge, linkActive=true, deviceId=27831dfb-fecb-48fc-b19f-bd11a48ac6cb, macAddr=00:1a:4a:32:3d:17, network=yorku}, {filter=vdsm-no-mac-spoofing, nicModel=pv, address={bus=0x00, domain=0x0000, function=0x0, slot=0x04, type=pci}, type=interface, specParams={inbound={}, outbound={}}, device=bridge, linkActive=true, deviceId=cb4529cf-db32-4c12-8aba-b4c98edd27aa, macAddr=00:1a:4a:32:3d:18, network=mrpriv}, {address={bus=0x00, domain=0x0000, function=0x2, slot=0x01, type=pci}, index=0, model=piix3-uhci, type=controller, specParams={}, device=usb, deviceId=b9604c74-45ec-44e9-b370-93b2d09d09fd}, {address={bus=0x00, domain=0x0000, function=0x0, slot=0x05, type=pci}, index=0, model=virtio-scsi, type=controller, specParams={}, device=scsi, deviceId=75e0ee00-a7bd-45e3-b0a4-9d99ba6a9946}, {address={bus=0x00, domain=0x0000, function=0x0, slot=0x06, type=pci}, type=controller, specParams={}, device=virtio-serial, deviceId=ffec31f9-0479-4c3a-9e97-15969a81cd4e}, {model=virtio, type=rng, specParams={source=urandom}, device=virtio, deviceId=dbcd3f11-c60d-439b-88f4-6501fc01bba6}], custom={device_5f45d2ad-18c0-4c9e-b2db-191071e278f5device_f996b11d-e7a3-43fb-9600-a5b5df2c7b6bdevice_ad99aae2-6ca2-4694-ad3d-634037642322=VmDevice:{id='VmDeviceId:{deviceId='ad99aae2-6ca2-4694-ad3d-634037642322', vmId='3e5a5728-8940-4fe7-b4af-11fb8e12f116'}', device='spicevmc', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=3}', managed='false', plugged='true', readOnly='false', deviceAlias='channel2', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_5f45d2ad-18c0-4c9e-b2db-191071e278f5device_f996b11d-e7a3-43fb-9600-a5b5df2c7b6b=VmDevice:{id='VmDeviceId:{deviceId='f996b11d-e7a3-43fb-9600-a5b5df2c7b6b', vmId='3e5a5728-8940-4fe7-b4af-11fb8e12f116'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=2}', managed='false', plugged='true', readOnly='false', deviceAlias='channel1', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_5f45d2ad-18c0-4c9e-b2db-191071e278f5device_f996b11d-e7a3-43fb-9600-a5b5df2c7b6bdevice_ad99aae2-6ca2-4694-ad3d-634037642322device_b865730b-1acd-47ad-9a6a-28c09906ad75=VmDevice:{id='VmDeviceId:{deviceId='b865730b-1acd-47ad-9a6a-28c09906ad75', vmId='3e5a5728-8940-4fe7-b4af-11fb8e12f116'}', device='ide', type='CONTROLLER', bootOrder='0', specParams='[]', address='{slot=0x01, bus=0x00, domain=0x0000, type=pci, function=0x1}', managed='false', plugged='true', readOnly='false', deviceAlias='ide', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_5f45d2ad-18c0-4c9e-b2db-191071e278f5=VmDevice:{id='VmDeviceId:{deviceId='5f45d2ad-18c0-4c9e-b2db-191071e278f5', vmId='3e5a5728-8940-4fe7-b4af-11fb8e12f116'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=1}', managed='false', plugged='true', readOnly='false', deviceAlias='channel0', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}}, display=qxl, timeOffset=0, nice=0, maxMemSize=16384, maxMemSlots=16, bootMenuEnable=false, memSize=4096} 2017-10-01 16:35:00,546-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] FINISH, CreateVDSCommand, log id: 51058938 2017-10-01 16:35:00,549-04 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] FINISH, CreateVmVDSCommand, return: WaitForLaunch, log id: 2f167898 2017-10-01 16:35:00,549-04 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] Lock freed to object 'EngineLock:{exclusiveLocks='[3e5a5728-8940-4fe7-b4af-11fb8e12f116=VM]', sharedLocks=''}' 2017-10-01 16:35:00,553-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] EVENT_ID: USER_STARTED_VM(153), Correlation ID: ef0a1d32-bc65-4a25-a600-995f2ad50edb, Job ID: 25019cf4-538e-47d7-902e-07157ab3b570, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM www was started by jas@eecs.yorku.ca-authz (Host: virt4). 2017-10-01 16:35:00,564-04 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] Running command: RunVmCommand internal: false. Entities affected : ID: 911d6883-607c-464e-b46b-8e6141bdd2de Type: VMAction group RUN_VM with role type USER 2017-10-01 16:35:00,649-04 INFO [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] START, UpdateVmDynamicDataVDSCommand( UpdateVmDynamicDataVDSCommandParameters:{runAsync='true', hostId='null', vmId='911d6883-607c-464e-b46b-8e6141bdd2de', vmDynamic='org.ovirt.engine.core.common.businessentities.VmDynamic@cd06f6ea'}), log id: 238a138c 2017-10-01 16:35:00,651-04 INFO [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] FINISH, UpdateVmDynamicDataVDSCommand, log id: 238a138c 2017-10-01 16:35:00,653-04 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] START, CreateVmVDSCommand( CreateVmVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vmId='911d6883-607c-464e-b46b-8e6141bdd2de', vm='VM [hdb]'}), log id: 5bee8759 2017-10-01 16:35:00,655-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] START, CreateVDSCommand(HostName = virt4, CreateVmVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vmId='911d6883-607c-464e-b46b-8e6141bdd2de', vm='VM [hdb]'}), log id: 1da2f3be 2017-10-01 16:35:00,658-04 INFO [org.ovirt.engine.core.vdsbroker.builder.vminfo.VmInfoBuilderImpl] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] Bootable disk '38312f6a-3afd-409c-b4f1-4b4a65b14618' set to index '0' 2017-10-01 16:35:00,672-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] VM {memGuaranteedSize=682, smpThreadsPerCore=1, cpuType=SandyBridge, vmId=911d6883-607c-464e-b46b-8e6141bdd2de, acpiEnable=true, vmType=kvm, smp=1, smpCoresPerSocket=1, emulatedMachine=pc-i440fx-rhel7.3.0, smartcardEnable=false, guestNumaNodes=[{memory=1024, cpus=0, nodeIndex=0}], transparentHugePages=true, displayNetwork=yorku, vmName=hdb, maxVCpus=16, kvmEnable=true, devices=[{address={bus=0x00, domain=0x0000, function=0x0, slot=0x02, type=pci}, type=video, specParams={heads=1, vram=32768}, device=qxl, deviceId=0e984ccf-23a3-43d5-bcef-b3454fc9d59b}, {type=graphics, specParams={fileTransferEnable=true, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,ssmartcard,susbredir, copyPasteEnable=true}, device=spice, deviceId=62ba37d1-71cb-4c67-ac3c-bab7f7cf57b2}, {iface=ide, shared=false, path=, address={bus=1, controller=0, unit=0, type=drive, target=0}, readonly=true, index=2, type=disk, specParams={path=}, device=cdrom, deviceId=19c953ed-14e0-41f1-a621-c29a0a27da87}, {discard=false, shared=false, address={bus=0x00, domain=0x0000, function=0x0, slot=0x07, type=pci}, imageID=38312f6a-3afd-409c-b4f1-4b4a65b14618, format=raw, index=0, optional=false, type=disk, deviceId=38312f6a-3afd-409c-b4f1-4b4a65b14618, domainID=c8653dba-63ec-4447-872c-0ffcc6af9c0f, propagateErrors=off, iface=virtio, readonly=false, bootOrder=1, poolID=aca61eac-ed8a-484b-bbee-3645148f6405, volumeID=3c55bcb2-ebc4-42f5-aa49-0a00f30eb864, specParams={}, device=disk}, {discard=false, shared=false, address={bus=0x00, domain=0x0000, function=0x0, slot=0x08, type=pci}, imageID=b251711a-409b-4f39-986b-0a68bf533a8b, format=raw, optional=false, type=disk, deviceId=b251711a-409b-4f39-986b-0a68bf533a8b, domainID=c8653dba-63ec-4447-872c-0ffcc6af9c0f, propagateErrors=off, iface=virtio, readonly=false, poolID=aca61eac-ed8a-484b-bbee-3645148f6405, volumeID=1c76d65b-81a6-40a0-a0d9-3fd3adeacda3, specParams={}, device=disk}, {filter=vdsm-no-mac-spoofing, nicModel=pv, address={bus=0x00, domain=0x0000, function=0x0, slot=0x03, type=pci}, type=interface, specParams={inbound={}, outbound={}}, device=bridge, linkActive=true, deviceId=97b4ee59-b06f-422b-9844-a6c48a7d0060, macAddr=00:1a:4a:32:3d:30, network=yorku}, {filter=vdsm-no-mac-spoofing, nicModel=pv, address={bus=0x00, domain=0x0000, function=0x0, slot=0x04, type=pci}, type=interface, specParams={inbound={}, outbound={}}, device=bridge, linkActive=true, deviceId=4cb353f0-642d-4bd1-bdc6-59a99fdb8809, macAddr=00:1a:4a:32:3d:31, network=mrpriv}, {address={bus=0x00, domain=0x0000, function=0x2, slot=0x01, type=pci}, index=0, model=piix3-uhci, type=controller, specParams={}, device=usb, deviceId=06bf46c1-54cd-429a-8d62-b3918a694c1e}, {address={bus=0x00, domain=0x0000, function=0x0, slot=0x05, type=pci}, index=0, model=virtio-scsi, type=controller, specParams={}, device=scsi, deviceId=17da955a-ddb6-4a51-ad1d-9f83cec24230}, {address={bus=0x00, domain=0x0000, function=0x0, slot=0x06, type=pci}, type=controller, specParams={}, device=virtio-serial, deviceId=da447f36-f260-4798-a34c-58397b0bf385}, {model=virtio, type=rng, specParams={source=urandom}, device=virtio, deviceId=03139496-e111-4f50-929d-cdc166bc95f2}], custom={device_200d4fa2-8a9a-43e5-b769-3420de44e63ddevice_ef6ba1cf-fdfb-4ec5-9b48-48d5e5ce8ad9device_10e4eeaa-a048-45db-baad-94fe84486584=VmDevice:{id='VmDeviceId:{deviceId='10e4eeaa-a048-45db-baad-94fe84486584', vmId='911d6883-607c-464e-b46b-8e6141bdd2de'}', device='spicevmc', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=3}', managed='false', plugged='true', readOnly='false', deviceAlias='channel2', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_200d4fa2-8a9a-43e5-b769-3420de44e63d=VmDevice:{id='VmDeviceId:{deviceId='200d4fa2-8a9a-43e5-b769-3420de44e63d', vmId='911d6883-607c-464e-b46b-8e6141bdd2de'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=1}', managed='false', plugged='true', readOnly='false', deviceAlias='channel0', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_200d4fa2-8a9a-43e5-b769-3420de44e63ddevice_ef6ba1cf-fdfb-4ec5-9b48-48d5e5ce8ad9device_10e4eeaa-a048-45db-baad-94fe84486584device_cadec10e-b41e-4470-a3b0-6bc21d303ba3=VmDevice:{id='VmDeviceId:{deviceId='cadec10e-b41e-4470-a3b0-6bc21d303ba3', vmId='911d6883-607c-464e-b46b-8e6141bdd2de'}', device='ide', type='CONTROLLER', bootOrder='0', specParams='[]', address='{slot=0x01, bus=0x00, domain=0x0000, type=pci, function=0x1}', managed='false', plugged='true', readOnly='false', deviceAlias='ide', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_200d4fa2-8a9a-43e5-b769-3420de44e63ddevice_ef6ba1cf-fdfb-4ec5-9b48-48d5e5ce8ad9=VmDevice:{id='VmDeviceId:{deviceId='ef6ba1cf-fdfb-4ec5-9b48-48d5e5ce8ad9', vmId='911d6883-607c-464e-b46b-8e6141bdd2de'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=2}', managed='false', plugged='true', readOnly='false', deviceAlias='channel1', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}}, display=qxl, timeOffset=0, nice=0, maxMemSize=4096, maxMemSlots=16, bootMenuEnable=false, memSize=1024} 2017-10-01 16:35:01,649-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] FINISH, CreateVDSCommand, log id: 1da2f3be 2017-10-01 16:35:01,652-04 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] FINISH, CreateVmVDSCommand, return: WaitForLaunch, log id: 5bee8759 2017-10-01 16:35:01,652-04 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] Lock freed to object 'EngineLock:{exclusiveLocks='[911d6883-607c-464e-b46b-8e6141bdd2de=VM]', sharedLocks=''}' 2017-10-01 16:35:01,656-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] EVENT_ID: USER_STARTED_VM(153), Correlation ID: ef0a1d32-bc65-4a25-a600-995f2ad50edb, Job ID: 81ea0e32-f2bf-4694-8683-f2115884f51f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM hdb was started by jas@eecs.yorku.ca-authz (Host: virt4). 2017-10-01 16:35:01,667-04 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] Running command: RunVmCommand internal: false. Entities affected : ID: c64d80f9-a3bd-44f3-89c4-f1b9b509cdde Type: VMAction group RUN_VM with role type USER 2017-10-01 16:35:01,753-04 INFO [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] START, UpdateVmDynamicDataVDSCommand( UpdateVmDynamicDataVDSCommandParameters:{runAsync='true', hostId='null', vmId='c64d80f9-a3bd-44f3-89c4-f1b9b509cdde', vmDynamic='org.ovirt.engine.core.common.businessentities.VmDynamic@5287733c'}), log id: 764f708d 2017-10-01 16:35:01,755-04 INFO [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] FINISH, UpdateVmDynamicDataVDSCommand, log id: 764f708d 2017-10-01 16:35:01,757-04 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] START, CreateVmVDSCommand( CreateVmVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vmId='c64d80f9-a3bd-44f3-89c4-f1b9b509cdde', vm='VM [red]'}), log id: 49215700 2017-10-01 16:35:01,758-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] START, CreateVDSCommand(HostName = virt1, CreateVmVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vmId='c64d80f9-a3bd-44f3-89c4-f1b9b509cdde', vm='VM [red]'}), log id: 4be9f911 2017-10-01 16:35:01,761-04 INFO [org.ovirt.engine.core.vdsbroker.builder.vminfo.VmInfoBuilderImpl] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] Bootable disk 'e5df3cc7-f88f-4419-897e-6e5e43d3872d' set to index '0' 2017-10-01 16:35:01,774-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] VM {memGuaranteedSize=16384, smpThreadsPerCore=1, cpuType=SandyBridge, vmId=c64d80f9-a3bd-44f3-89c4-f1b9b509cdde, acpiEnable=true, vmType=kvm, smp=6, smpCoresPerSocket=1, emulatedMachine=pc-i440fx-rhel7.3.0, smartcardEnable=false, guestNumaNodes=[{memory=20480, cpus=0,1,2,3,4,5, nodeIndex=0}], transparentHugePages=true, displayNetwork=yorku, vmName=red, maxVCpus=16, kvmEnable=true, devices=[{address={bus=0x00, domain=0x0000, function=0x0, slot=0x02, type=pci}, type=video, specParams={heads=1, vram=32768}, device=qxl, deviceId=8731e9cf-5da0-4ef5-8b2d-22c42637aaf9}, {type=graphics, specParams={fileTransferEnable=true, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,ssmartcard,susbredir, copyPasteEnable=true}, device=spice, deviceId=0000001f-001f-001f-001f-000000000268}, {iface=ide, shared=false, path=, address={bus=1, controller=0, unit=0, type=drive, target=0}, readonly=true, index=2, type=disk, specParams={path=}, device=cdrom, deviceId=c8381907-ce0f-463c-a649-f7b25453eb2b}, {discard=false, shared=false, address={bus=0, controller=0, unit=0, type=drive, target=0}, imageID=e5df3cc7-f88f-4419-897e-6e5e43d3872d, format=raw, index=0, optional=false, type=disk, deviceId=e5df3cc7-f88f-4419-897e-6e5e43d3872d, domainID=c8653dba-63ec-4447-872c-0ffcc6af9c0f, propagateErrors=off, iface=scsi, readonly=false, bootOrder=1, poolID=aca61eac-ed8a-484b-bbee-3645148f6405, volumeID=b568dee6-5db5-44a6-b523-550b5e446605, specParams={}, device=disk}, {filter=vdsm-no-mac-spoofing, nicModel=pv, address={bus=0x00, domain=0x0000, function=0x0, slot=0x03, type=pci}, type=interface, specParams={inbound={}, outbound={}}, device=bridge, linkActive=true, deviceId=ef5d0b67-d3c8-4367-ba72-8de265d278b3, macAddr=00:1a:4a:32:3d:02, network=yorku}, {filter=vdsm-no-mac-spoofing, nicModel=pv, address={bus=0x00, domain=0x0000, function=0x0, slot=0x04, type=pci}, type=interface, specParams={inbound={}, outbound={}}, device=bridge, linkActive=true, deviceId=a6e35579-c489-4c28-a3a3-3635847d8bc3, macAddr=00:1a:4a:32:3d:03, network=mrpriv}, {address={bus=0x00, domain=0x0000, function=0x2, slot=0x01, type=pci}, index=0, model=piix3-uhci, type=controller, specParams={}, device=usb, deviceId=0cdaa218-4309-4421-8ba4-cbb54a07f770}, {address={bus=0x00, domain=0x0000, function=0x0, slot=0x05, type=pci}, index=0, model=virtio-scsi, type=controller, specParams={}, device=scsi, deviceId=fa8dcf83-ad47-426c-97ab-718d34ff1873}, {address={bus=0x00, domain=0x0000, function=0x0, slot=0x06, type=pci}, type=controller, specParams={}, device=virtio-serial, deviceId=ba8cf08c-e732-47e9-a4be-5e746f21c139}, {model=virtio, type=rng, specParams={source=urandom}, device=virtio, deviceId=e6cc13e5-c664-4435-a187-46d2b34ea993}], custom={device_16dad27f-1884-4f32-8736-6bbaf8cd2fbddevice_d9f2931e-e8e0-4b83-8534-9c35daabff71device_28c1e908-e275-411a-bbeb-2920a3032e74device_44b628f3-6129-417b-a46d-de82d1b20d76=VmDevice:{id='VmDeviceId:{deviceId='44b628f3-6129-417b-a46d-de82d1b20d76', vmId='c64d80f9-a3bd-44f3-89c4-f1b9b509cdde'}', device='ide', type='CONTROLLER', bootOrder='0', specParams='[]', address='{slot=0x01, bus=0x00, domain=0x0000, type=pci, function=0x1}', managed='false', plugged='true', readOnly='false', deviceAlias='ide', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_16dad27f-1884-4f32-8736-6bbaf8cd2fbddevice_d9f2931e-e8e0-4b83-8534-9c35daabff71device_28c1e908-e275-411a-bbeb-2920a3032e74=VmDevice:{id='VmDeviceId:{deviceId='28c1e908-e275-411a-bbeb-2920a3032e74', vmId='c64d80f9-a3bd-44f3-89c4-f1b9b509cdde'}', device='spicevmc', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=3}', managed='false', plugged='true', readOnly='false', deviceAlias='channel2', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_16dad27f-1884-4f32-8736-6bbaf8cd2fbd=VmDevice:{id='VmDeviceId:{deviceId='16dad27f-1884-4f32-8736-6bbaf8cd2fbd', vmId='c64d80f9-a3bd-44f3-89c4-f1b9b509cdde'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=1}', managed='false', plugged='true', readOnly='false', deviceAlias='channel0', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_16dad27f-1884-4f32-8736-6bbaf8cd2fbddevice_d9f2931e-e8e0-4b83-8534-9c35daabff71=VmDevice:{id='VmDeviceId:{deviceId='d9f2931e-e8e0-4b83-8534-9c35daabff71', vmId='c64d80f9-a3bd-44f3-89c4-f1b9b509cdde'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=2}', managed='false', plugged='true', readOnly='false', deviceAlias='channel1', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}}, display=qxl, timeOffset=0, nice=0, maxMemSize=81920, maxMemSlots=16, bootMenuEnable=false, memSize=20480} 2017-10-01 16:35:02,056-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] FINISH, CreateVDSCommand, log id: 4be9f911 2017-10-01 16:35:02,058-04 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] FINISH, CreateVmVDSCommand, return: WaitForLaunch, log id: 49215700 2017-10-01 16:35:02,059-04 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] Lock freed to object 'EngineLock:{exclusiveLocks='[c64d80f9-a3bd-44f3-89c4-f1b9b509cdde=VM]', sharedLocks=''}' 2017-10-01 16:35:02,063-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] EVENT_ID: USER_STARTED_VM(153), Correlation ID: ef0a1d32-bc65-4a25-a600-995f2ad50edb, Job ID: 5ba74392-51d3-44bd-83b8-b068d5c38233, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM red was started by jas@eecs.yorku.ca-authz (Host: virt1). 2017-10-01 16:35:02,087-04 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] Running command: RunVmCommand internal: false. Entities affected : ID: 64734fca-4d6c-443c-82d1-abf7cf106601 Type: VMAction group RUN_VM with role type USER 2017-10-01 16:35:02,170-04 INFO [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] START, UpdateVmDynamicDataVDSCommand( UpdateVmDynamicDataVDSCommandParameters:{runAsync='true', hostId='null', vmId='64734fca-4d6c-443c-82d1-abf7cf106601', vmDynamic='org.ovirt.engine.core.common.businessentities.VmDynamic@5f518824'}), log id: 700a8502 2017-10-01 16:35:02,171-04 INFO [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] FINISH, UpdateVmDynamicDataVDSCommand, log id: 700a8502 2017-10-01 16:35:02,173-04 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] START, CreateVmVDSCommand( CreateVmVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vmId='64734fca-4d6c-443c-82d1-abf7cf106601', vm='VM [webdav]'}), log id: 75fab305 2017-10-01 16:35:02,175-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] START, CreateVDSCommand(HostName = virt4, CreateVmVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vmId='64734fca-4d6c-443c-82d1-abf7cf106601', vm='VM [webdav]'}), log id: cf08fb2 2017-10-01 16:35:02,178-04 INFO [org.ovirt.engine.core.vdsbroker.builder.vminfo.VmInfoBuilderImpl] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] Bootable disk 'ec77b6c9-87e6-4886-979a-945d05767d03' set to index '0' 2017-10-01 16:35:02,196-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] VM {memGuaranteedSize=682, smpThreadsPerCore=1, cpuType=SandyBridge, vmId=64734fca-4d6c-443c-82d1-abf7cf106601, acpiEnable=true, vmType=kvm, smp=1, smpCoresPerSocket=1, emulatedMachine=pc-i440fx-rhel7.3.0, smartcardEnable=false, guestNumaNodes=[{memory=1024, cpus=0, nodeIndex=0}], transparentHugePages=true, displayNetwork=yorku, vmName=webdav, maxVCpus=16, kvmEnable=true, devices=[{address={bus=0x00, domain=0x0000, function=0x0, slot=0x02, type=pci}, type=video, specParams={heads=1, vram=32768}, device=qxl, deviceId=ee3d0719-91d1-41ab-9552-4b3ca4aa427a}, {type=graphics, specParams={fileTransferEnable=true, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,ssmartcard,susbredir, copyPasteEnable=true}, device=spice, deviceId=00000021-0021-0021-0021-00000000022e}, {iface=ide, shared=false, path=, address={bus=1, controller=0, unit=0, type=drive, target=0}, readonly=true, index=2, type=disk, specParams={path=}, device=cdrom, deviceId=254c4475-ea71-4e6b-9c95-4acb162cd3f0}, {discard=false, shared=false, address={bus=0x00, domain=0x0000, function=0x0, slot=0x07, type=pci}, imageID=ec77b6c9-87e6-4886-979a-945d05767d03, format=raw, index=0, optional=false, type=disk, deviceId=ec77b6c9-87e6-4886-979a-945d05767d03, domainID=c8653dba-63ec-4447-872c-0ffcc6af9c0f, propagateErrors=off, iface=virtio, readonly=false, bootOrder=1, poolID=aca61eac-ed8a-484b-bbee-3645148f6405, volumeID=7ac5f512-da36-40c0-a798-57369280dfbe, specParams={}, device=disk}, {discard=false, shared=false, address={bus=0x00, domain=0x0000, function=0x0, slot=0x08, type=pci}, imageID=00d67eef-b095-4db4-a2bd-7ac3db618a36, format=raw, optional=false, type=disk, deviceId=00d67eef-b095-4db4-a2bd-7ac3db618a36, domainID=c8653dba-63ec-4447-872c-0ffcc6af9c0f, propagateErrors=off, iface=virtio, readonly=false, poolID=aca61eac-ed8a-484b-bbee-3645148f6405, volumeID=d63967cb-0fc1-4856-b05b-38f034fbee4a, specParams={}, device=disk}, {filter=vdsm-no-mac-spoofing, nicModel=pv, address={bus=0x00, domain=0x0000, function=0x0, slot=0x03, type=pci}, type=interface, specParams={inbound={}, outbound={}}, device=bridge, linkActive=true, deviceId=05efc9be-02ee-4789-82e9-6dc628b88805, macAddr=00:1a:4a:32:3d:23, network=yorku}, {filter=vdsm-no-mac-spoofing, nicModel=pv, address={bus=0x00, domain=0x0000, function=0x0, slot=0x04, type=pci}, type=interface, specParams={inbound={}, outbound={}}, device=bridge, linkActive=true, deviceId=1df93294-942e-4b18-bda9-b569a13982e2, macAddr=00:1a:4a:32:3d:24, network=mrpriv}, {address={bus=0x00, domain=0x0000, function=0x2, slot=0x01, type=pci}, index=0, model=piix3-uhci, type=controller, specParams={}, device=usb, deviceId=a9c7d04c-b190-4569-942a-0dc43767191c}, {address={bus=0x00, function=0x0, domain=0x0000, slot=0x09, type=pci}, type=balloon, specParams={model=virtio}, device=memballoon, deviceId=cb9f42c3-d97c-440f-a7c3-91bd2e910cd5}, {address={bus=0x00, domain=0x0000, function=0x0, slot=0x05, type=pci}, index=0, model=virtio-scsi, type=controller, specParams={}, device=scsi, deviceId=2b01df54-45a6-4a89-82d1-c0438c4751b7}, {address={bus=0x00, domain=0x0000, function=0x0, slot=0x06, type=pci}, type=controller, specParams={}, device=virtio-serial, deviceId=32999d33-d6b1-4d18-a899-b8cbd376b63e}, {model=virtio, type=rng, specParams={source=urandom}, device=virtio, deviceId=6ebf73e6-42da-46d0-9bdd-b621c53d8150}], custom={device_729e6b0e-f43d-48a8-8407-2b824a5bf38cdevice_94e2e5e8-291a-4354-9c28-82e9c5e921afdevice_87531847-0176-468a-b1e3-f2fc0c53ced6=VmDevice:{id='VmDeviceId:{deviceId='87531847-0176-468a-b1e3-f2fc0c53ced6', vmId='64734fca-4d6c-443c-82d1-abf7cf106601'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=2}', managed='false', plugged='true', readOnly='false', deviceAlias='channel1', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_729e6b0e-f43d-48a8-8407-2b824a5bf38cdevice_94e2e5e8-291a-4354-9c28-82e9c5e921af=VmDevice:{id='VmDeviceId:{deviceId='94e2e5e8-291a-4354-9c28-82e9c5e921af', vmId='64734fca-4d6c-443c-82d1-abf7cf106601'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=1}', managed='false', plugged='true', readOnly='false', deviceAlias='channel0', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_729e6b0e-f43d-48a8-8407-2b824a5bf38c=VmDevice:{id='VmDeviceId:{deviceId='729e6b0e-f43d-48a8-8407-2b824a5bf38c', vmId='64734fca-4d6c-443c-82d1-abf7cf106601'}', device='ide', type='CONTROLLER', bootOrder='0', specParams='[]', address='{slot=0x01, bus=0x00, domain=0x0000, type=pci, function=0x1}', managed='false', plugged='true', readOnly='false', deviceAlias='ide', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_729e6b0e-f43d-48a8-8407-2b824a5bf38cdevice_94e2e5e8-291a-4354-9c28-82e9c5e921afdevice_87531847-0176-468a-b1e3-f2fc0c53ced6device_4e280ea3-c260-4154-b63a-0779beaff05f=VmDevice:{id='VmDeviceId:{deviceId='4e280ea3-c260-4154-b63a-0779beaff05f', vmId='64734fca-4d6c-443c-82d1-abf7cf106601'}', device='spicevmc', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=3}', managed='false', plugged='true', readOnly='false', deviceAlias='channel2', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}}, display=qxl, timeOffset=0, nice=0, maxMemSize=4096, maxMemSlots=16, bootMenuEnable=false, memSize=1024} 2017-10-01 16:35:03,039-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] FINISH, CreateVDSCommand, log id: cf08fb2 2017-10-01 16:35:03,042-04 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] FINISH, CreateVmVDSCommand, return: WaitForLaunch, log id: 75fab305 2017-10-01 16:35:03,042-04 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] Lock freed to object 'EngineLock:{exclusiveLocks='[64734fca-4d6c-443c-82d1-abf7cf106601=VM]', sharedLocks=''}' 2017-10-01 16:35:03,046-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] EVENT_ID: USER_STARTED_VM(153), Correlation ID: ef0a1d32-bc65-4a25-a600-995f2ad50edb, Job ID: e6f0835c-d631-4459-87cd-ef560eaf3ab2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM webdav was started by jas@eecs.yorku.ca-authz (Host: virt4). 2017-10-01 16:35:03,056-04 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] Running command: RunVmCommand internal: false. Entities affected : ID: 59287796-c634-4f3c-94b1-2c9699986637 Type: VMAction group RUN_VM with role type USER 2017-10-01 16:35:03,140-04 INFO [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] START, UpdateVmDynamicDataVDSCommand( UpdateVmDynamicDataVDSCommandParameters:{runAsync='true', hostId='null', vmId='59287796-c634-4f3c-94b1-2c9699986637', vmDynamic='org.ovirt.engine.core.common.businessentities.VmDynamic@d4971a06'}), log id: 46198ca3 2017-10-01 16:35:03,142-04 INFO [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] FINISH, UpdateVmDynamicDataVDSCommand, log id: 46198ca3 2017-10-01 16:35:03,144-04 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] START, CreateVmVDSCommand( CreateVmVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vmId='59287796-c634-4f3c-94b1-2c9699986637', vm='VM [vpn1]'}), log id: 1a8cb2fa 2017-10-01 16:35:03,145-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] START, CreateVDSCommand(HostName = virt3, CreateVmVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vmId='59287796-c634-4f3c-94b1-2c9699986637', vm='VM [vpn1]'}), log id: 7d42ae50 2017-10-01 16:35:03,148-04 INFO [org.ovirt.engine.core.vdsbroker.builder.vminfo.VmInfoBuilderImpl] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] Bootable disk '0f4dcba2-fc56-4ffc-9993-f5d2dc8e6049' set to index '0' 2017-10-01 16:35:03,160-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] VM {memGuaranteedSize=1365, smpThreadsPerCore=1, cpuType=SandyBridge, vmId=59287796-c634-4f3c-94b1-2c9699986637, acpiEnable=true, vmType=kvm, smp=2, smpCoresPerSocket=1, emulatedMachine=pc-i440fx-rhel7.3.0, smartcardEnable=false, guestNumaNodes=[{memory=2048, cpus=0,1, nodeIndex=0}], transparentHugePages=true, displayNetwork=yorku, vmName=vpn1, maxVCpus=16, kvmEnable=true, devices=[{address={bus=0x00, domain=0x0000, function=0x0, slot=0x02, type=pci}, type=video, specParams={vgamem=16384, heads=1, ram=65536, vram=8192}, device=qxl, deviceId=51beb361-aadb-4836-bbea-a60f48083398}, {type=graphics, specParams={fileTransferEnable=true, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,ssmartcard,susbredir, copyPasteEnable=true}, device=spice, deviceId=22da7b7b-8a47-4039-b926-d18882ea2f9d}, {iface=ide, shared=false, path=, address={bus=1, controller=0, unit=0, type=drive, target=0}, readonly=true, index=2, type=disk, specParams={path=}, device=cdrom, deviceId=6ccf1509-e04a-4ddc-ac83-6583565b08a3}, {discard=false, shared=false, address={bus=0x00, domain=0x0000, function=0x0, slot=0x07, type=pci}, imageID=0f4dcba2-fc56-4ffc-9993-f5d2dc8e6049, format=raw, index=0, optional=false, type=disk, deviceId=0f4dcba2-fc56-4ffc-9993-f5d2dc8e6049, domainID=c8653dba-63ec-4447-872c-0ffcc6af9c0f, propagateErrors=off, iface=virtio, readonly=false, bootOrder=1, poolID=aca61eac-ed8a-484b-bbee-3645148f6405, volumeID=5cca6758-dc53-441a-873a-cb1ff2f2e961, specParams={}, device=disk}, {filter=vdsm-no-mac-spoofing, nicModel=pv, address={bus=0x00, domain=0x0000, function=0x0, slot=0x03, type=pci}, type=interface, specParams={inbound={}, outbound={}}, device=bridge, linkActive=true, deviceId=309bc8fa-7f37-4f0b-9f25-b8b63d1f92af, macAddr=00:1a:4a:32:3d:06, network=yorku}, {filter=vdsm-no-mac-spoofing, nicModel=pv, address={bus=0x00, domain=0x0000, function=0x0, slot=0x04, type=pci}, type=interface, specParams={inbound={}, outbound={}}, device=bridge, linkActive=true, deviceId=c0f2f436-054e-4eeb-9e11-0b4fd4023971, macAddr=00:1a:4a:32:3d:1f, network=mrpriv}, {address={bus=0x00, domain=0x0000, function=0x2, slot=0x01, type=pci}, index=0, model=piix3-uhci, type=controller, specParams={}, device=usb, deviceId=23683193-a2e8-4d84-a55c-80f9a18f3e2f}, {address={bus=0x00, function=0x0, domain=0x0000, slot=0x08, type=pci}, type=balloon, specParams={model=virtio}, device=memballoon, deviceId=6ec155e4-bdd3-42b5-8011-00c3c7ba2148}, {address={bus=0x00, domain=0x0000, function=0x0, slot=0x05, type=pci}, index=0, model=virtio-scsi, type=controller, specParams={}, device=scsi, deviceId=4a7f9149-0573-4ded-ab36-5ea20c736be6}, {address={bus=0x00, domain=0x0000, function=0x0, slot=0x06, type=pci}, type=controller, specParams={}, device=virtio-serial, deviceId=e0784650-0821-400c-a215-bc5d6b6c70b0}, {model=virtio, type=rng, specParams={source=urandom}, device=virtio, deviceId=ee3afe40-d284-4dca-ade1-a68da6b7e4ee}], custom={device_69e94312-87d3-493e-9665-92afdfe15039device_10796316-52cc-4731-9300-9f72ef554368=VmDevice:{id='VmDeviceId:{deviceId='10796316-52cc-4731-9300-9f72ef554368', vmId='59287796-c634-4f3c-94b1-2c9699986637'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=2}', managed='false', plugged='true', readOnly='false', deviceAlias='channel1', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_69e94312-87d3-493e-9665-92afdfe15039device_10796316-52cc-4731-9300-9f72ef554368device_5034296d-1ccc-4781-9aa0-f0677e556ed6=VmDevice:{id='VmDeviceId:{deviceId='5034296d-1ccc-4781-9aa0-f0677e556ed6', vmId='59287796-c634-4f3c-94b1-2c9699986637'}', device='spicevmc', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=3}', managed='false', plugged='true', readOnly='false', deviceAlias='channel2', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_69e94312-87d3-493e-9665-92afdfe15039device_10796316-52cc-4731-9300-9f72ef554368device_5034296d-1ccc-4781-9aa0-f0677e556ed6device_1ef0ef51-10e7-436c-9fb7-dd73bd03cf40=VmDevice:{id='VmDeviceId:{deviceId='1ef0ef51-10e7-436c-9fb7-dd73bd03cf40', vmId='59287796-c634-4f3c-94b1-2c9699986637'}', device='ide', type='CONTROLLER', bootOrder='0', specParams='[]', address='{slot=0x01, bus=0x00, domain=0x0000, type=pci, function=0x1}', managed='false', plugged='true', readOnly='false', deviceAlias='ide', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_69e94312-87d3-493e-9665-92afdfe15039=VmDevice:{id='VmDeviceId:{deviceId='69e94312-87d3-493e-9665-92afdfe15039', vmId='59287796-c634-4f3c-94b1-2c9699986637'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=1}', managed='false', plugged='true', readOnly='false', deviceAlias='channel0', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}}, display=qxl, timeOffset=0, nice=0, maxMemSize=8192, maxMemSlots=16, bootMenuEnable=false, memSize=2048} 2017-10-01 16:35:03,330-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] FINISH, CreateVDSCommand, log id: 7d42ae50 2017-10-01 16:35:03,332-04 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] FINISH, CreateVmVDSCommand, return: WaitForLaunch, log id: 1a8cb2fa 2017-10-01 16:35:03,332-04 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] Lock freed to object 'EngineLock:{exclusiveLocks='[59287796-c634-4f3c-94b1-2c9699986637=VM]', sharedLocks=''}' 2017-10-01 16:35:03,335-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] EVENT_ID: USER_STARTED_VM(153), Correlation ID: ef0a1d32-bc65-4a25-a600-995f2ad50edb, Job ID: 9224dd2d-47bb-4619-bdd4-9ef43dc27c5b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM vpn1 was started by jas@eecs.yorku.ca-authz (Host: virt3). 2017-10-01 16:35:03,346-04 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] Running command: RunVmCommand internal: false. Entities affected : ID: 3bb83ad5-dc00-445d-8708-2db18c53e0e6 Type: VMAction group RUN_VM with role type USER 2017-10-01 16:35:03,429-04 INFO [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] START, UpdateVmDynamicDataVDSCommand( UpdateVmDynamicDataVDSCommandParameters:{runAsync='true', hostId='null', vmId='3bb83ad5-dc00-445d-8708-2db18c53e0e6', vmDynamic='org.ovirt.engine.core.common.businessentities.VmDynamic@bdf2d336'}), log id: 6a50105c 2017-10-01 16:35:03,431-04 INFO [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] FINISH, UpdateVmDynamicDataVDSCommand, log id: 6a50105c 2017-10-01 16:35:03,433-04 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] START, CreateVmVDSCommand( CreateVmVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vmId='3bb83ad5-dc00-445d-8708-2db18c53e0e6', vm='VM [www-vhost]'}), log id: 525b8bdb 2017-10-01 16:35:03,434-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] START, CreateVDSCommand(HostName = virt3, CreateVmVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vmId='3bb83ad5-dc00-445d-8708-2db18c53e0e6', vm='VM [www-vhost]'}), log id: 41576bdf 2017-10-01 16:35:03,437-04 INFO [org.ovirt.engine.core.vdsbroker.builder.vminfo.VmInfoBuilderImpl] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] Bootable disk '2664778d-81e8-4c3e-acae-d6d1c3f52435' set to index '0' 2017-10-01 16:35:03,449-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] VM {memGuaranteedSize=4096, smpThreadsPerCore=1, cpuType=SandyBridge, vmId=3bb83ad5-dc00-445d-8708-2db18c53e0e6, acpiEnable=true, vmType=kvm, smp=4, smpCoresPerSocket=1, emulatedMachine=pc-i440fx-rhel7.3.0, smartcardEnable=false, guestNumaNodes=[{memory=4096, cpus=0,1,2,3, nodeIndex=0}], transparentHugePages=true, displayNetwork=yorku, vmName=www-vhost, maxVCpus=16, kvmEnable=true, devices=[{address={bus=0x00, domain=0x0000, function=0x0, slot=0x02, type=pci}, type=video, specParams={heads=1, vram=32768}, device=qxl, deviceId=38d13453-d17a-45d0-82ee-c4eeaac3d114}, {type=graphics, specParams={fileTransferEnable=true, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,ssmartcard,susbredir, copyPasteEnable=true}, device=spice, deviceId=0000002b-002b-002b-002b-000000000192}, {iface=ide, shared=false, path=, address={bus=1, controller=0, unit=0, type=drive, target=0}, readonly=true, index=2, type=disk, specParams={path=}, device=cdrom, deviceId=3e5c164c-9728-4a90-9827-794db063d337}, {discard=false, shared=false, address={bus=0x00, domain=0x0000, function=0x0, slot=0x07, type=pci}, imageID=2664778d-81e8-4c3e-acae-d6d1c3f52435, format=raw, index=0, optional=false, type=disk, deviceId=2664778d-81e8-4c3e-acae-d6d1c3f52435, domainID=c8653dba-63ec-4447-872c-0ffcc6af9c0f, propagateErrors=off, iface=virtio, readonly=false, bootOrder=1, poolID=aca61eac-ed8a-484b-bbee-3645148f6405, volumeID=733d5370-ea70-445f-8a1a-91718e199fd0, specParams={}, device=disk}, {discard=false, shared=false, address={bus=0x00, domain=0x0000, function=0x0, slot=0x08, type=pci}, imageID=fd6f6d4a-92fc-4305-829e-8d877a875b21, format=raw, optional=false, type=disk, deviceId=fd6f6d4a-92fc-4305-829e-8d877a875b21, domainID=c8653dba-63ec-4447-872c-0ffcc6af9c0f, propagateErrors=off, iface=virtio, readonly=false, poolID=aca61eac-ed8a-484b-bbee-3645148f6405, volumeID=13ba0973-661a-44c0-8123-dd74c7b33415, specParams={}, device=disk}, {filter=vdsm-no-mac-spoofing, nicModel=pv, address={bus=0x00, domain=0x0000, function=0x0, slot=0x03, type=pci}, type=interface, specParams={inbound={}, outbound={}}, device=bridge, linkActive=true, deviceId=05c92c7c-cdd7-4fbe-ba5d-9a66fadd3a96, macAddr=00:1a:4a:32:3d:19, network=yorku}, {filter=vdsm-no-mac-spoofing, nicModel=pv, address={bus=0x00, domain=0x0000, function=0x0, slot=0x04, type=pci}, type=interface, specParams={inbound={}, outbound={}}, device=bridge, linkActive=true, deviceId=7de2b8e7-6023-419e-89d1-80e591fb1be2, macAddr=00:1a:4a:32:3d:1a, network=mrpriv}, {address={bus=0x00, domain=0x0000, function=0x2, slot=0x01, type=pci}, index=0, model=piix3-uhci, type=controller, specParams={}, device=usb, deviceId=5a0fe599-4a6c-4e0b-b05d-c54eedb9c5c0}, {address={bus=0x00, domain=0x0000, function=0x0, slot=0x05, type=pci}, index=0, model=virtio-scsi, type=controller, specParams={}, device=scsi, deviceId=88d88a6b-5e42-4465-8c85-ddad9ec1b20e}, {address={bus=0x00, domain=0x0000, function=0x0, slot=0x06, type=pci}, type=controller, specParams={}, device=virtio-serial, deviceId=f32a3d2f-e0c9-4bc7-b552-4338caed084d}, {model=virtio, type=rng, specParams={source=urandom}, device=virtio, deviceId=40077ad2-763a-4b41-83c0-040c8d68a97f}], custom={device_53c485bb-d380-4a20-b5cb-cc26b577713fdevice_11c0a69b-6d65-4b6f-8a60-955245903452=VmDevice:{id='VmDeviceId:{deviceId='11c0a69b-6d65-4b6f-8a60-955245903452', vmId='3bb83ad5-dc00-445d-8708-2db18c53e0e6'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=2}', managed='false', plugged='true', readOnly='false', deviceAlias='channel1', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_53c485bb-d380-4a20-b5cb-cc26b577713fdevice_11c0a69b-6d65-4b6f-8a60-955245903452device_e2ed8694-2c47-49a9-ada0-1f7e225e55f1device_8eb8899b-6d1c-4062-b558-3c39463747d9=VmDevice:{id='VmDeviceId:{deviceId='8eb8899b-6d1c-4062-b558-3c39463747d9', vmId='3bb83ad5-dc00-445d-8708-2db18c53e0e6'}', device='ide', type='CONTROLLER', bootOrder='0', specParams='[]', address='{slot=0x01, bus=0x00, domain=0x0000, type=pci, function=0x1}', managed='false', plugged='true', readOnly='false', deviceAlias='ide', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_53c485bb-d380-4a20-b5cb-cc26b577713fdevice_11c0a69b-6d65-4b6f-8a60-955245903452device_e2ed8694-2c47-49a9-ada0-1f7e225e55f1=VmDevice:{id='VmDeviceId:{deviceId='e2ed8694-2c47-49a9-ada0-1f7e225e55f1', vmId='3bb83ad5-dc00-445d-8708-2db18c53e0e6'}', device='spicevmc', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=3}', managed='false', plugged='true', readOnly='false', deviceAlias='channel2', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_53c485bb-d380-4a20-b5cb-cc26b577713f=VmDevice:{id='VmDeviceId:{deviceId='53c485bb-d380-4a20-b5cb-cc26b577713f', vmId='3bb83ad5-dc00-445d-8708-2db18c53e0e6'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=1}', managed='false', plugged='true', readOnly='false', deviceAlias='channel0', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}}, display=qxl, timeOffset=0, nice=0, maxMemSize=16384, maxMemSlots=16, bootMenuEnable=false, memSize=4096} 2017-10-01 16:35:03,631-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] FINISH, CreateVDSCommand, log id: 41576bdf 2017-10-01 16:35:03,634-04 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] FINISH, CreateVmVDSCommand, return: WaitForLaunch, log id: 525b8bdb 2017-10-01 16:35:03,634-04 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] Lock freed to object 'EngineLock:{exclusiveLocks='[3bb83ad5-dc00-445d-8708-2db18c53e0e6=VM]', sharedLocks=''}' 2017-10-01 16:35:03,638-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] EVENT_ID: USER_STARTED_VM(153), Correlation ID: ef0a1d32-bc65-4a25-a600-995f2ad50edb, Job ID: b56dc0f1-596d-4d10-a246-4ca9c36dad9b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM www-vhost was started by jas@eecs.yorku.ca-authz (Host: virt3). 2017-10-01 16:35:03,648-04 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] Running command: RunVmCommand internal: false. Entities affected : ID: cc1dbe93-bb80-451e-8b3b-c9cc5e90432f Type: VMAction group RUN_VM with role type USER 2017-10-01 16:35:03,732-04 INFO [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] START, UpdateVmDynamicDataVDSCommand( UpdateVmDynamicDataVDSCommandParameters:{runAsync='true', hostId='null', vmId='cc1dbe93-bb80-451e-8b3b-c9cc5e90432f', vmDynamic='org.ovirt.engine.core.common.businessentities.VmDynamic@62e97143'}), log id: 51ac2d48 2017-10-01 16:35:03,733-04 INFO [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] FINISH, UpdateVmDynamicDataVDSCommand, log id: 51ac2d48 2017-10-01 16:35:03,736-04 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] START, CreateVmVDSCommand( CreateVmVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vmId='cc1dbe93-bb80-451e-8b3b-c9cc5e90432f', vm='VM [license]'}), log id: 6f89f2d4 2017-10-01 16:35:03,737-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] START, CreateVDSCommand(HostName = virt3, CreateVmVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vmId='cc1dbe93-bb80-451e-8b3b-c9cc5e90432f', vm='VM [license]'}), log id: 32125af8 2017-10-01 16:35:03,740-04 INFO [org.ovirt.engine.core.vdsbroker.builder.vminfo.VmInfoBuilderImpl] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] Bootable disk 'c082b403-cff2-480b-bd43-be62b31dda63' set to index '0' 2017-10-01 16:35:03,751-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] VM {memGuaranteedSize=2730, smpThreadsPerCore=1, cpuType=SandyBridge, vmId=cc1dbe93-bb80-451e-8b3b-c9cc5e90432f, acpiEnable=true, vmType=kvm, smp=2, smpCoresPerSocket=1, emulatedMachine=pc-i440fx-rhel7.3.0, smartcardEnable=false, guestNumaNodes=[{memory=4096, cpus=0,1, nodeIndex=0}], transparentHugePages=true, displayNetwork=yorku, vmName=license, maxVCpus=16, kvmEnable=true, devices=[{address={bus=0x00, domain=0x0000, function=0x0, slot=0x02, type=pci}, type=video, specParams={heads=1, vram=32768}, device=qxl, deviceId=d5a30fdb-906b-40bd-a9f6-8bb0e526240d}, {type=graphics, specParams={fileTransferEnable=true, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,ssmartcard,susbredir, copyPasteEnable=true}, device=spice, deviceId=6fd350b3-e6f6-46ec-981d-d0250aebb024}, {iface=ide, shared=false, path=, address={bus=1, controller=0, unit=0, type=drive, target=0}, readonly=true, index=2, type=disk, specParams={path=}, device=cdrom, deviceId=695fc832-2777-4c7a-afb8-45f01dff92e6}, {discard=false, shared=false, address={bus=0x00, domain=0x0000, function=0x0, slot=0x07, type=pci}, imageID=c082b403-cff2-480b-bd43-be62b31dda63, format=raw, index=0, optional=false, type=disk, deviceId=c082b403-cff2-480b-bd43-be62b31dda63, domainID=c8653dba-63ec-4447-872c-0ffcc6af9c0f, propagateErrors=off, iface=virtio, readonly=false, bootOrder=1, poolID=aca61eac-ed8a-484b-bbee-3645148f6405, volumeID=c75f5834-cc52-4397-a36f-e268a6f59598, specParams={}, device=disk}, {discard=false, shared=false, address={bus=0x00, domain=0x0000, function=0x0, slot=0x09, type=pci}, imageID=a27b5225-0edf-4285-bade-0d48706fc8d9, format=raw, optional=false, type=disk, deviceId=a27b5225-0edf-4285-bade-0d48706fc8d9, domainID=c8653dba-63ec-4447-872c-0ffcc6af9c0f, propagateErrors=off, iface=virtio, readonly=false, poolID=aca61eac-ed8a-484b-bbee-3645148f6405, volumeID=119dd202-651f-498b-8ba1-1ebc6cb20507, specParams={}, device=disk}, {filter=vdsm-no-mac-spoofing, nicModel=pv, address={bus=0x00, domain=0x0000, function=0x0, slot=0x03, type=pci}, type=interface, specParams={inbound={}, outbound={}}, device=bridge, linkActive=true, deviceId=7bb2ddf2-eae4-4d83-b216-31e47934ecb1, macAddr=00:15:17:63:3a:b5, network=yorku}, {filter=vdsm-no-mac-spoofing, nicModel=pv, address={bus=0x00, domain=0x0000, function=0x0, slot=0x04, type=pci}, type=interface, specParams={inbound={}, outbound={}}, device=bridge, linkActive=true, deviceId=1c90d3bf-edf2-4d19-b3b6-181291948845, macAddr=00:1a:4a:32:3d:20, network=mrpriv}, {address={bus=0x00, domain=0x0000, function=0x2, slot=0x01, type=pci}, index=0, model=piix3-uhci, type=controller, specParams={}, device=usb, deviceId=49398d3c-ad96-429d-89be-695ae6a5c3c9}, {address={bus=0x00, domain=0x0000, function=0x0, slot=0x05, type=pci}, index=0, model=virtio-scsi, type=controller, specParams={}, device=scsi, deviceId=312c5f82-2789-49b5-9321-30838cbd9d10}, {address={bus=0x00, domain=0x0000, function=0x0, slot=0x06, type=pci}, type=controller, specParams={}, device=virtio-serial, deviceId=064f3655-afb9-4691-8183-635a2d3d62fc}, {model=virtio, type=rng, specParams={source=urandom}, device=virtio, deviceId=9ff9398e-f277-4513-8fb9-d68060156ed4}], custom={device_c6badb18-1587-4042-8dbb-08b45a345f53device_cf1ac984-b593-4de6-8b5a-332e8f931f50device_024fe828-a608-42f1-8e96-6425c35b0690device_b1e2b19c-73b7-494d-9186-12d039cc81ca=VmDevice:{id='VmDeviceId:{deviceId='b1e2b19c-73b7-494d-9186-12d039cc81ca', vmId='cc1dbe93-bb80-451e-8b3b-c9cc5e90432f'}', device='ide', type='CONTROLLER', bootOrder='0', specParams='[]', address='{slot=0x01, bus=0x00, domain=0x0000, type=pci, function=0x1}', managed='false', plugged='true', readOnly='false', deviceAlias='ide', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_c6badb18-1587-4042-8dbb-08b45a345f53device_cf1ac984-b593-4de6-8b5a-332e8f931f50=VmDevice:{id='VmDeviceId:{deviceId='cf1ac984-b593-4de6-8b5a-332e8f931f50', vmId='cc1dbe93-bb80-451e-8b3b-c9cc5e90432f'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=2}', managed='false', plugged='true', readOnly='false', deviceAlias='channel1', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_c6badb18-1587-4042-8dbb-08b45a345f53device_cf1ac984-b593-4de6-8b5a-332e8f931f50device_024fe828-a608-42f1-8e96-6425c35b0690=VmDevice:{id='VmDeviceId:{deviceId='024fe828-a608-42f1-8e96-6425c35b0690', vmId='cc1dbe93-bb80-451e-8b3b-c9cc5e90432f'}', device='spicevmc', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=3}', managed='false', plugged='true', readOnly='false', deviceAlias='channel2', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_c6badb18-1587-4042-8dbb-08b45a345f53=VmDevice:{id='VmDeviceId:{deviceId='c6badb18-1587-4042-8dbb-08b45a345f53', vmId='cc1dbe93-bb80-451e-8b3b-c9cc5e90432f'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=1}', managed='false', plugged='true', readOnly='false', deviceAlias='channel0', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}}, display=qxl, timeOffset=0, nice=0, maxMemSize=16384, maxMemSlots=16, bootMenuEnable=false, memSize=4096} 2017-10-01 16:35:03,768-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] FINISH, CreateVDSCommand, log id: 32125af8 2017-10-01 16:35:03,777-04 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] FINISH, CreateVmVDSCommand, return: WaitForLaunch, log id: 6f89f2d4 2017-10-01 16:35:03,777-04 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] Lock freed to object 'EngineLock:{exclusiveLocks='[cc1dbe93-bb80-451e-8b3b-c9cc5e90432f=VM]', sharedLocks=''}' 2017-10-01 16:35:03,780-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] EVENT_ID: USER_STARTED_VM(153), Correlation ID: ef0a1d32-bc65-4a25-a600-995f2ad50edb, Job ID: 92c99462-6311-4bc7-9199-ed531cf806fa, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM license was started by jas@eecs.yorku.ca-authz (Host: virt3). 2017-10-01 16:35:03,790-04 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] Running command: RunVmCommand internal: false. Entities affected : ID: 9ea2821e-557f-45cb-9872-1e6c5aeaf246 Type: VMAction group RUN_VM with role type USER 2017-10-01 16:35:03,874-04 INFO [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] START, UpdateVmDynamicDataVDSCommand( UpdateVmDynamicDataVDSCommandParameters:{runAsync='true', hostId='null', vmId='9ea2821e-557f-45cb-9872-1e6c5aeaf246', vmDynamic='org.ovirt.engine.core.common.businessentities.VmDynamic@76dc6494'}), log id: 2b274a09 2017-10-01 16:35:03,875-04 INFO [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] FINISH, UpdateVmDynamicDataVDSCommand, log id: 2b274a09 2017-10-01 16:35:03,877-04 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] START, CreateVmVDSCommand( CreateVmVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vmId='9ea2821e-557f-45cb-9872-1e6c5aeaf246', vm='VM [rs]'}), log id: 32c3ce26 2017-10-01 16:35:03,878-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] START, CreateVDSCommand(HostName = virt2, CreateVmVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vmId='9ea2821e-557f-45cb-9872-1e6c5aeaf246', vm='VM [rs]'}), log id: 4ff20733 2017-10-01 16:35:03,881-04 INFO [org.ovirt.engine.core.vdsbroker.builder.vminfo.VmInfoBuilderImpl] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] Bootable disk '7f80d948-ba64-4473-95a1-2bb1a03eddf8' set to index '0' 2017-10-01 16:35:03,893-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] VM {memGuaranteedSize=682, smpThreadsPerCore=1, cpuType=SandyBridge, vmId=9ea2821e-557f-45cb-9872-1e6c5aeaf246, acpiEnable=true, vmType=kvm, smp=1, smpCoresPerSocket=1, emulatedMachine=pc-i440fx-rhel7.3.0, smartcardEnable=false, guestNumaNodes=[{memory=1024, cpus=0, nodeIndex=0}], transparentHugePages=true, displayNetwork=yorku, vmName=rs, maxVCpus=16, kvmEnable=true, devices=[{address={bus=0x00, domain=0x0000, function=0x0, slot=0x02, type=pci}, type=video, specParams={heads=1, vram=32768}, device=qxl, deviceId=50ec4546-11c6-4076-91a2-d718e5184f0c}, {type=graphics, specParams={fileTransferEnable=true, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,ssmartcard,susbredir, copyPasteEnable=true}, device=spice, deviceId=58c8cadc-6a62-488c-85d2-dc06c41907f3}, {iface=ide, shared=false, path=, address={bus=1, controller=0, unit=0, type=drive, target=0}, readonly=true, index=2, type=disk, specParams={path=}, device=cdrom, deviceId=7099bc22-a7d7-48aa-921a-7ece00cf9aba}, {discard=false, shared=false, address={bus=0x00, domain=0x0000, function=0x0, slot=0x07, type=pci}, imageID=7f80d948-ba64-4473-95a1-2bb1a03eddf8, format=raw, index=0, optional=false, type=disk, deviceId=7f80d948-ba64-4473-95a1-2bb1a03eddf8, domainID=c8653dba-63ec-4447-872c-0ffcc6af9c0f, propagateErrors=off, iface=virtio, readonly=false, bootOrder=1, poolID=aca61eac-ed8a-484b-bbee-3645148f6405, volumeID=6bacbc93-6066-4fbd-bb01-30c004ad9f7a, specParams={}, device=disk}, {filter=vdsm-no-mac-spoofing, nicModel=pv, address={bus=0x00, domain=0x0000, function=0x0, slot=0x03, type=pci}, type=interface, specParams={inbound={}, outbound={}}, device=bridge, linkActive=true, deviceId=06b53504-ff99-457d-97ef-930e0a1160bb, macAddr=00:1a:4a:32:3d:25, network=yorku}, {filter=vdsm-no-mac-spoofing, nicModel=pv, address={bus=0x00, domain=0x0000, function=0x0, slot=0x04, type=pci}, type=interface, specParams={inbound={}, outbound={}}, device=bridge, linkActive=true, deviceId=b2de3dd6-7f6b-4f1c-bc24-72a239c4095b, macAddr=00:1a:4a:32:3d:26, network=mrpriv}, {address={bus=0x00, domain=0x0000, function=0x2, slot=0x01, type=pci}, index=0, model=piix3-uhci, type=controller, specParams={}, device=usb, deviceId=e23b00c2-f244-4691-86da-77c5992e6017}, {address={bus=0x00, domain=0x0000, function=0x0, slot=0x05, type=pci}, index=0, model=virtio-scsi, type=controller, specParams={}, device=scsi, deviceId=9cef35f4-14f3-45f9-b9fd-c401a6f20371}, {address={bus=0x00, domain=0x0000, function=0x0, slot=0x06, type=pci}, type=controller, specParams={}, device=virtio-serial, deviceId=ab35ea50-65cc-436a-bb17-85d0d8ee40c7}, {model=virtio, type=rng, specParams={source=urandom}, device=virtio, deviceId=0915ea55-b389-4808-883e-cad240bfa869}], custom={device_ede985f5-ebdf-4058-9bd6-85b32f6086d1device_210d38b2-63a2-4b1e-8d59-3ede51837fc8=VmDevice:{id='VmDeviceId:{deviceId='210d38b2-63a2-4b1e-8d59-3ede51837fc8', vmId='9ea2821e-557f-45cb-9872-1e6c5aeaf246'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=2}', managed='false', plugged='true', readOnly='false', deviceAlias='channel1', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_ede985f5-ebdf-4058-9bd6-85b32f6086d1device_210d38b2-63a2-4b1e-8d59-3ede51837fc8device_b674e26e-33c4-45f6-b6d6-a751958e7889device_64356cea-121e-49a6-b2e3-b61cf6ef581d=VmDevice:{id='VmDeviceId:{deviceId='64356cea-121e-49a6-b2e3-b61cf6ef581d', vmId='9ea2821e-557f-45cb-9872-1e6c5aeaf246'}', device='ide', type='CONTROLLER', bootOrder='0', specParams='[]', address='{slot=0x01, bus=0x00, domain=0x0000, type=pci, function=0x1}', managed='false', plugged='true', readOnly='false', deviceAlias='ide', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_ede985f5-ebdf-4058-9bd6-85b32f6086d1=VmDevice:{id='VmDeviceId:{deviceId='ede985f5-ebdf-4058-9bd6-85b32f6086d1', vmId='9ea2821e-557f-45cb-9872-1e6c5aeaf246'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=1}', managed='false', plugged='true', readOnly='false', deviceAlias='channel0', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_ede985f5-ebdf-4058-9bd6-85b32f6086d1device_210d38b2-63a2-4b1e-8d59-3ede51837fc8device_b674e26e-33c4-45f6-b6d6-a751958e7889=VmDevice:{id='VmDeviceId:{deviceId='b674e26e-33c4-45f6-b6d6-a751958e7889', vmId='9ea2821e-557f-45cb-9872-1e6c5aeaf246'}', device='spicevmc', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=3}', managed='false', plugged='true', readOnly='false', deviceAlias='channel2', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}}, display=qxl, timeOffset=0, nice=0, maxMemSize=4096, maxMemSlots=16, bootMenuEnable=false, memSize=1024} 2017-10-01 16:35:04,670-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] FINISH, CreateVDSCommand, log id: 4ff20733 2017-10-01 16:35:04,673-04 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] FINISH, CreateVmVDSCommand, return: WaitForLaunch, log id: 32c3ce26 2017-10-01 16:35:04,673-04 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] Lock freed to object 'EngineLock:{exclusiveLocks='[9ea2821e-557f-45cb-9872-1e6c5aeaf246=VM]', sharedLocks=''}' 2017-10-01 16:35:04,678-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] EVENT_ID: USER_STARTED_VM(153), Correlation ID: ef0a1d32-bc65-4a25-a600-995f2ad50edb, Job ID: 4e144a04-558f-4e61-8910-5c153de6e0ba, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM rs was started by jas@eecs.yorku.ca-authz (Host: virt2). 2017-10-01 16:35:04,688-04 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] Running command: RunVmCommand internal: false. Entities affected : ID: ae56c31c-71f7-433a-b198-a1c2c846325e Type: VMAction group RUN_VM with role type USER 2017-10-01 16:35:04,772-04 INFO [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] START, UpdateVmDynamicDataVDSCommand( UpdateVmDynamicDataVDSCommandParameters:{runAsync='true', hostId='null', vmId='ae56c31c-71f7-433a-b198-a1c2c846325e', vmDynamic='org.ovirt.engine.core.common.businessentities.VmDynamic@c4a0aeb3'}), log id: 5bdd7500 2017-10-01 16:35:04,774-04 INFO [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] FINISH, UpdateVmDynamicDataVDSCommand, log id: 5bdd7500 2017-10-01 16:35:04,776-04 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] START, CreateVmVDSCommand( CreateVmVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vmId='ae56c31c-71f7-433a-b198-a1c2c846325e', vm='VM [forum]'}), log id: 2ca91742 2017-10-01 16:35:04,777-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] START, CreateVDSCommand(HostName = virt4, CreateVmVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vmId='ae56c31c-71f7-433a-b198-a1c2c846325e', vm='VM [forum]'}), log id: 365dbae9 2017-10-01 16:35:04,781-04 INFO [org.ovirt.engine.core.vdsbroker.builder.vminfo.VmInfoBuilderImpl] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] Bootable disk '27a1aca7-69cf-4418-94b6-2b6470cbfe41' set to index '0' 2017-10-01 16:35:04,793-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] VM {memGuaranteedSize=4096, smpThreadsPerCore=1, cpuType=SandyBridge, vmId=ae56c31c-71f7-433a-b198-a1c2c846325e, acpiEnable=true, vmType=kvm, smp=2, smpCoresPerSocket=1, emulatedMachine=pc-i440fx-rhel7.3.0, smartcardEnable=false, guestNumaNodes=[{memory=4096, cpus=0,1, nodeIndex=0}], transparentHugePages=true, displayNetwork=yorku, vmName=forum, maxVCpus=16, kvmEnable=true, devices=[{address={bus=0x00, domain=0x0000, function=0x0, slot=0x02, type=pci}, type=video, specParams={heads=1, vram=32768}, device=qxl, deviceId=c5b6d4cb-6019-4b47-8985-7bcead81e50e}, {type=graphics, specParams={fileTransferEnable=true, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,ssmartcard,susbredir, copyPasteEnable=true}, device=spice, deviceId=00000030-0030-0030-0030-0000000001cd}, {iface=ide, shared=false, path=, address={bus=1, controller=0, unit=0, type=drive, target=0}, readonly=true, index=2, type=disk, specParams={path=}, device=cdrom, deviceId=ebc8b070-b242-436e-94e1-67cd78d35c08}, {discard=false, shared=false, address={bus=0x00, domain=0x0000, function=0x0, slot=0x07, type=pci}, imageID=27a1aca7-69cf-4418-94b6-2b6470cbfe41, format=raw, index=0, optional=false, type=disk, deviceId=27a1aca7-69cf-4418-94b6-2b6470cbfe41, domainID=c8653dba-63ec-4447-872c-0ffcc6af9c0f, propagateErrors=off, iface=virtio, readonly=false, bootOrder=1, poolID=aca61eac-ed8a-484b-bbee-3645148f6405, volumeID=1481e3bb-c62d-4c5f-ba74-c30ffc511186, specParams={}, device=disk}, {discard=false, shared=false, address={bus=0x00, domain=0x0000, function=0x0, slot=0x08, type=pci}, imageID=c333d24b-dd23-401a-8739-c5a4854a93f3, format=raw, optional=false, type=disk, deviceId=c333d24b-dd23-401a-8739-c5a4854a93f3, domainID=c8653dba-63ec-4447-872c-0ffcc6af9c0f, propagateErrors=off, iface=virtio, readonly=false, poolID=aca61eac-ed8a-484b-bbee-3645148f6405, volumeID=9ecb430f-fa4a-45c5-b6c3-cd26c62ff677, specParams={}, device=disk}, {filter=vdsm-no-mac-spoofing, nicModel=pv, address={bus=0x00, domain=0x0000, function=0x0, slot=0x03, type=pci}, type=interface, specParams={inbound={}, outbound={}}, device=bridge, linkActive=true, deviceId=f9d101c7-74a0-4754-b029-729330112e6d, macAddr=00:1a:4a:32:3d:13, network=yorku}, {filter=vdsm-no-mac-spoofing, nicModel=pv, address={bus=0x00, domain=0x0000, function=0x0, slot=0x04, type=pci}, type=interface, specParams={inbound={}, outbound={}}, device=bridge, linkActive=true, deviceId=be661c39-8e60-4957-b68b-276e6b88d5e6, macAddr=00:1a:4a:32:3d:14, network=mrpriv}, {address={bus=0x00, domain=0x0000, function=0x2, slot=0x01, type=pci}, index=0, model=piix3-uhci, type=controller, specParams={}, device=usb, deviceId=bcfaa8b0-d96e-4d38-adf3-9bb545b0a67e}, {address={bus=0x00, domain=0x0000, function=0x0, slot=0x05, type=pci}, index=0, model=virtio-scsi, type=controller, specParams={}, device=scsi, deviceId=a3604be1-1cb1-4807-bc4c-58f403319a30}, {address={bus=0x00, domain=0x0000, function=0x0, slot=0x06, type=pci}, type=controller, specParams={}, device=virtio-serial, deviceId=54f5814d-f151-40ae-a15a-a54ee1eded94}, {model=virtio, type=rng, specParams={source=urandom}, device=virtio, deviceId=9ae72c8c-e2a9-4810-83c6-1549ffa10ad8}], custom={device_80a61fdb-b84a-4c66-bbcc-4a3a257b583fdevice_5f697df1-cc6b-402f-8b25-ad143b5ee816device_f660a99d-ec1f-4a44-8f99-ae21606bfd34=VmDevice:{id='VmDeviceId:{deviceId='f660a99d-ec1f-4a44-8f99-ae21606bfd34', vmId='ae56c31c-71f7-433a-b198-a1c2c846325e'}', device='spicevmc', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=3}', managed='false', plugged='true', readOnly='false', deviceAlias='channel2', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_80a61fdb-b84a-4c66-bbcc-4a3a257b583fdevice_5f697df1-cc6b-402f-8b25-ad143b5ee816=VmDevice:{id='VmDeviceId:{deviceId='5f697df1-cc6b-402f-8b25-ad143b5ee816', vmId='ae56c31c-71f7-433a-b198-a1c2c846325e'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=2}', managed='false', plugged='true', readOnly='false', deviceAlias='channel1', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_80a61fdb-b84a-4c66-bbcc-4a3a257b583fdevice_5f697df1-cc6b-402f-8b25-ad143b5ee816device_f660a99d-ec1f-4a44-8f99-ae21606bfd34device_62c8af1e-b59b-4bd8-99dd-40948e53b6df=VmDevice:{id='VmDeviceId:{deviceId='62c8af1e-b59b-4bd8-99dd-40948e53b6df', vmId='ae56c31c-71f7-433a-b198-a1c2c846325e'}', device='ide', type='CONTROLLER', bootOrder='0', specParams='[]', address='{slot=0x01, bus=0x00, domain=0x0000, type=pci, function=0x1}', managed='false', plugged='true', readOnly='false', deviceAlias='ide', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_80a61fdb-b84a-4c66-bbcc-4a3a257b583f=VmDevice:{id='VmDeviceId:{deviceId='80a61fdb-b84a-4c66-bbcc-4a3a257b583f', vmId='ae56c31c-71f7-433a-b198-a1c2c846325e'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=1}', managed='false', plugged='true', readOnly='false', deviceAlias='channel0', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}}, display=qxl, timeOffset=0, nice=0, maxMemSize=16384, maxMemSlots=16, bootMenuEnable=false, memSize=4096} 2017-10-01 16:35:04,947-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] FINISH, CreateVDSCommand, log id: 365dbae9 2017-10-01 16:35:04,950-04 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] FINISH, CreateVmVDSCommand, return: WaitForLaunch, log id: 2ca91742 2017-10-01 16:35:04,950-04 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] Lock freed to object 'EngineLock:{exclusiveLocks='[ae56c31c-71f7-433a-b198-a1c2c846325e=VM]', sharedLocks=''}' 2017-10-01 16:35:04,954-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] EVENT_ID: USER_STARTED_VM(153), Correlation ID: ef0a1d32-bc65-4a25-a600-995f2ad50edb, Job ID: c826c447-fa76-4929-9d86-edd94dea51df, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM forum was started by jas@eecs.yorku.ca-authz (Host: virt4). 2017-10-01 16:35:04,974-04 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] Running command: RunVmCommand internal: false. Entities affected : ID: d66235b4-7c8d-4da2-a572-82ecc9c53a59 Type: VMAction group RUN_VM with role type USER 2017-10-01 16:35:05,058-04 INFO [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] START, UpdateVmDynamicDataVDSCommand( UpdateVmDynamicDataVDSCommandParameters:{runAsync='true', hostId='null', vmId='d66235b4-7c8d-4da2-a572-82ecc9c53a59', vmDynamic='org.ovirt.engine.core.common.businessentities.VmDynamic@204fd306'}), log id: 1219ac2d 2017-10-01 16:35:05,060-04 INFO [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] FINISH, UpdateVmDynamicDataVDSCommand, log id: 1219ac2d 2017-10-01 16:35:05,062-04 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] START, CreateVmVDSCommand( CreateVmVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vmId='d66235b4-7c8d-4da2-a572-82ecc9c53a59', vm='VM [www-las]'}), log id: 6369f92 2017-10-01 16:35:05,064-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] START, CreateVDSCommand(HostName = virt3, CreateVmVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vmId='d66235b4-7c8d-4da2-a572-82ecc9c53a59', vm='VM [www-las]'}), log id: 5e587936 2017-10-01 16:35:05,067-04 INFO [org.ovirt.engine.core.vdsbroker.builder.vminfo.VmInfoBuilderImpl] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] Bootable disk '1ddc83e8-35d8-42d0-882a-d8bc35c2efa2' set to index '0' 2017-10-01 16:35:05,081-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] VM {memGuaranteedSize=4096, smpThreadsPerCore=1, cpuType=SandyBridge, vmId=d66235b4-7c8d-4da2-a572-82ecc9c53a59, acpiEnable=true, vmType=kvm, smp=4, smpCoresPerSocket=1, emulatedMachine=pc-i440fx-rhel7.3.0, smartcardEnable=false, guestNumaNodes=[{memory=4096, cpus=0,1,2,3, nodeIndex=0}], transparentHugePages=true, displayNetwork=yorku, vmName=www-las, maxVCpus=16, kvmEnable=true, devices=[{address={bus=0x00, domain=0x0000, function=0x0, slot=0x02, type=pci}, type=video, specParams={heads=1, vram=32768}, device=qxl, deviceId=807d13fa-945a-492f-9bd9-00dda83650b3}, {type=graphics, specParams={fileTransferEnable=true, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,ssmartcard,susbredir, copyPasteEnable=true}, device=spice, deviceId=00000032-0032-0032-0032-0000000002c8}, {iface=ide, shared=false, path=, address={bus=1, controller=0, unit=0, type=drive, target=0}, readonly=true, index=2, type=disk, specParams={path=}, device=cdrom, deviceId=33931b19-35b1-4848-91d7-177e45ac7bee}, {discard=false, shared=false, address={bus=0x00, domain=0x0000, function=0x0, slot=0x07, type=pci}, imageID=1ddc83e8-35d8-42d0-882a-d8bc35c2efa2, format=raw, index=0, optional=false, type=disk, deviceId=1ddc83e8-35d8-42d0-882a-d8bc35c2efa2, domainID=c8653dba-63ec-4447-872c-0ffcc6af9c0f, propagateErrors=off, iface=virtio, readonly=false, bootOrder=1, poolID=aca61eac-ed8a-484b-bbee-3645148f6405, volumeID=a59c8ccc-6535-4720-80ff-92aca6b54be0, specParams={}, device=disk}, {discard=false, shared=false, address={bus=0x00, domain=0x0000, function=0x0, slot=0x08, type=pci}, imageID=aa973707-8311-4111-8f5b-1de610736bec, format=raw, optional=false, type=disk, deviceId=aa973707-8311-4111-8f5b-1de610736bec, domainID=c8653dba-63ec-4447-872c-0ffcc6af9c0f, propagateErrors=off, iface=virtio, readonly=false, poolID=aca61eac-ed8a-484b-bbee-3645148f6405, volumeID=9a634692-1302-471f-a92e-c978b2b67fd0, specParams={}, device=disk}, {filter=vdsm-no-mac-spoofing, nicModel=pv, address={bus=0x00, domain=0x0000, function=0x0, slot=0x03, type=pci}, type=interface, specParams={inbound={}, outbound={}}, device=bridge, linkActive=true, deviceId=e413fb88-ed8b-41b2-b0d0-805dd9cc38ea, macAddr=00:1a:4a:32:3d:15, network=yorku}, {filter=vdsm-no-mac-spoofing, nicModel=pv, address={bus=0x00, domain=0x0000, function=0x0, slot=0x04, type=pci}, type=interface, specParams={inbound={}, outbound={}}, device=bridge, linkActive=true, deviceId=0c13d576-ca04-42cd-bf7a-8c72af3099ae, macAddr=00:1a:4a:32:3d:16, network=mrpriv}, {address={bus=0x00, domain=0x0000, function=0x2, slot=0x01, type=pci}, index=0, model=piix3-uhci, type=controller, specParams={}, device=usb, deviceId=75dba8d3-3783-47b6-95ea-7587c028d9d9}, {address={bus=0x00, domain=0x0000, function=0x0, slot=0x05, type=pci}, index=0, model=virtio-scsi, type=controller, specParams={}, device=scsi, deviceId=879f410a-f62e-44b9-8e34-8666e5aaf087}, {address={bus=0x00, domain=0x0000, function=0x0, slot=0x06, type=pci}, type=controller, specParams={}, device=virtio-serial, deviceId=f85a1fef-5b32-46ab-aafc-194bd2a2df30}, {model=virtio, type=rng, specParams={source=urandom}, device=virtio, deviceId=4440cf64-af9a-4390-a6eb-504a8c03a12a}], custom={device_e6f104af-4245-4a6d-8c1d-ecc3120daaabdevice_783850c4-36cd-49c5-b1b7-1b94ad859083=VmDevice:{id='VmDeviceId:{deviceId='783850c4-36cd-49c5-b1b7-1b94ad859083', vmId='d66235b4-7c8d-4da2-a572-82ecc9c53a59'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=2}', managed='false', plugged='true', readOnly='false', deviceAlias='channel1', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_e6f104af-4245-4a6d-8c1d-ecc3120daaab=VmDevice:{id='VmDeviceId:{deviceId='e6f104af-4245-4a6d-8c1d-ecc3120daaab', vmId='d66235b4-7c8d-4da2-a572-82ecc9c53a59'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=1}', managed='false', plugged='true', readOnly='false', deviceAlias='channel0', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_e6f104af-4245-4a6d-8c1d-ecc3120daaabdevice_783850c4-36cd-49c5-b1b7-1b94ad859083device_d94441b2-fa26-4878-b6d1-468c3d8cc29e=VmDevice:{id='VmDeviceId:{deviceId='d94441b2-fa26-4878-b6d1-468c3d8cc29e', vmId='d66235b4-7c8d-4da2-a572-82ecc9c53a59'}', device='spicevmc', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=3}', managed='false', plugged='true', readOnly='false', deviceAlias='channel2', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_e6f104af-4245-4a6d-8c1d-ecc3120daaabdevice_783850c4-36cd-49c5-b1b7-1b94ad859083device_d94441b2-fa26-4878-b6d1-468c3d8cc29edevice_905471fd-92ff-4c8a-ae24-c8b018279497=VmDevice:{id='VmDeviceId:{deviceId='905471fd-92ff-4c8a-ae24-c8b018279497', vmId='d66235b4-7c8d-4da2-a572-82ecc9c53a59'}', device='ide', type='CONTROLLER', bootOrder='0', specParams='[]', address='{slot=0x01, bus=0x00, domain=0x0000, type=pci, function=0x1}', managed='false', plugged='true', readOnly='false', deviceAlias='ide', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}}, display=qxl, timeOffset=0, nice=0, maxMemSize=16384, maxMemSlots=16, bootMenuEnable=false, memSize=4096} 2017-10-01 16:35:05,193-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] FINISH, CreateVDSCommand, log id: 5e587936 2017-10-01 16:35:05,196-04 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] FINISH, CreateVmVDSCommand, return: WaitForLaunch, log id: 6369f92 2017-10-01 16:35:05,196-04 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] Lock freed to object 'EngineLock:{exclusiveLocks='[d66235b4-7c8d-4da2-a572-82ecc9c53a59=VM]', sharedLocks=''}' 2017-10-01 16:35:05,200-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] EVENT_ID: USER_STARTED_VM(153), Correlation ID: ef0a1d32-bc65-4a25-a600-995f2ad50edb, Job ID: a7def337-f3fb-455e-babd-44b13883cb8a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM www-las was started by jas@eecs.yorku.ca-authz (Host: virt3). 2017-10-01 16:35:05,214-04 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] Running command: RunVmCommand internal: false. Entities affected : ID: 1734fed5-fc97-4714-bb63-55e8f4eb6d34 Type: VMAction group RUN_VM with role type USER 2017-10-01 16:35:05,298-04 INFO [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] START, UpdateVmDynamicDataVDSCommand( UpdateVmDynamicDataVDSCommandParameters:{runAsync='true', hostId='null', vmId='1734fed5-fc97-4714-bb63-55e8f4eb6d34', vmDynamic='org.ovirt.engine.core.common.businessentities.VmDynamic@28289edb'}), log id: 7c9c60ad 2017-10-01 16:35:05,300-04 INFO [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] FINISH, UpdateVmDynamicDataVDSCommand, log id: 7c9c60ad 2017-10-01 16:35:05,302-04 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] START, CreateVmVDSCommand( CreateVmVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vmId='1734fed5-fc97-4714-bb63-55e8f4eb6d34', vm='VM [wiki]'}), log id: 39507290 2017-10-01 16:35:05,303-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] START, CreateVDSCommand(HostName = virt2, CreateVmVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vmId='1734fed5-fc97-4714-bb63-55e8f4eb6d34', vm='VM [wiki]'}), log id: 19dc4d4d 2017-10-01 16:35:05,307-04 INFO [org.ovirt.engine.core.vdsbroker.builder.vminfo.VmInfoBuilderImpl] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] Bootable disk '4096159b-990c-49d0-81fb-4cd934f00c47' set to index '0' 2017-10-01 16:35:05,319-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] VM {memGuaranteedSize=4096, smpThreadsPerCore=1, cpuType=SandyBridge, vmId=1734fed5-fc97-4714-bb63-55e8f4eb6d34, acpiEnable=true, vmType=kvm, smp=2, smpCoresPerSocket=1, emulatedMachine=pc-i440fx-rhel7.3.0, smartcardEnable=false, guestNumaNodes=[{memory=4096, cpus=0,1, nodeIndex=0}], transparentHugePages=true, displayNetwork=yorku, vmName=wiki, maxVCpus=16, kvmEnable=true, devices=[{address={bus=0x00, domain=0x0000, function=0x0, slot=0x02, type=pci}, type=video, specParams={heads=1, vram=32768}, device=qxl, deviceId=ccde18db-7091-4bc3-bbee-a8fab06696d5}, {type=graphics, specParams={fileTransferEnable=true, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,ssmartcard,susbredir, copyPasteEnable=true}, device=spice, deviceId=0000002d-002d-002d-002d-0000000001ac}, {iface=ide, shared=false, path=, address={bus=1, controller=0, unit=0, type=drive, target=0}, readonly=true, index=2, type=disk, specParams={path=}, device=cdrom, deviceId=9f75bd5a-318a-41ae-8cad-dcf457a2a1ed}, {discard=false, shared=false, address={bus=0x00, domain=0x0000, function=0x0, slot=0x07, type=pci}, imageID=4096159b-990c-49d0-81fb-4cd934f00c47, format=raw, index=0, optional=false, type=disk, deviceId=4096159b-990c-49d0-81fb-4cd934f00c47, domainID=c8653dba-63ec-4447-872c-0ffcc6af9c0f, propagateErrors=off, iface=virtio, readonly=false, bootOrder=1, poolID=aca61eac-ed8a-484b-bbee-3645148f6405, volumeID=e58de177-2cfb-42fe-8884-2de3794056db, specParams={}, device=disk}, {discard=false, shared=false, address={bus=0x00, domain=0x0000, function=0x0, slot=0x08, type=pci}, imageID=fafeface-d302-4c3f-97b8-941b6bc3545f, format=raw, optional=false, type=disk, deviceId=fafeface-d302-4c3f-97b8-941b6bc3545f, domainID=c8653dba-63ec-4447-872c-0ffcc6af9c0f, propagateErrors=off, iface=virtio, readonly=false, poolID=aca61eac-ed8a-484b-bbee-3645148f6405, volumeID=53fc926a-8037-47b9-b7ee-da86132a4c78, specParams={}, device=disk}, {filter=vdsm-no-mac-spoofing, nicModel=pv, address={bus=0x00, domain=0x0000, function=0x0, slot=0x03, type=pci}, type=interface, specParams={inbound={}, outbound={}}, device=bridge, linkActive=true, deviceId=f88584d1-8f34-4bd6-a4b5-def75771b329, macAddr=00:1a:4a:32:3d:07, network=yorku}, {filter=vdsm-no-mac-spoofing, nicModel=pv, address={bus=0x00, domain=0x0000, function=0x0, slot=0x04, type=pci}, type=interface, specParams={inbound={}, outbound={}}, device=bridge, linkActive=true, deviceId=a3f3c6f5-2862-42bf-ad3e-305089af1034, macAddr=00:1a:4a:32:3d:08, network=mrpriv}, {address={bus=0x00, domain=0x0000, function=0x2, slot=0x01, type=pci}, index=0, model=piix3-uhci, type=controller, specParams={}, device=usb, deviceId=62494988-ca9d-4d45-ac6b-31b07c4de816}, {address={bus=0x00, domain=0x0000, function=0x0, slot=0x05, type=pci}, index=0, model=virtio-scsi, type=controller, specParams={}, device=scsi, deviceId=81b8187a-8ff3-4f9d-b989-7cb05efd5205}, {address={bus=0x00, domain=0x0000, function=0x0, slot=0x06, type=pci}, type=controller, specParams={}, device=virtio-serial, deviceId=24498793-8cfa-4f4c-be94-bbc0ef12f88f}, {model=virtio, type=rng, specParams={source=urandom}, device=virtio, deviceId=ae6c7e6e-b9d4-4b03-8593-1f0e6adbf707}], custom={device_9d496570-3467-475b-ba9c-3111478eb9a7device_fb1efd5f-e403-44f3-ae46-9e817b19ee91=VmDevice:{id='VmDeviceId:{deviceId='fb1efd5f-e403-44f3-ae46-9e817b19ee91', vmId='1734fed5-fc97-4714-bb63-55e8f4eb6d34'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=2}', managed='false', plugged='true', readOnly='false', deviceAlias='channel1', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_9d496570-3467-475b-ba9c-3111478eb9a7=VmDevice:{id='VmDeviceId:{deviceId='9d496570-3467-475b-ba9c-3111478eb9a7', vmId='1734fed5-fc97-4714-bb63-55e8f4eb6d34'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=1}', managed='false', plugged='true', readOnly='false', deviceAlias='channel0', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_9d496570-3467-475b-ba9c-3111478eb9a7device_fb1efd5f-e403-44f3-ae46-9e817b19ee91device_953c69bc-7182-496a-8180-1a8dc3230464device_9a7950a3-fb9b-4f82-adac-d8246f775af0=VmDevice:{id='VmDeviceId:{deviceId='9a7950a3-fb9b-4f82-adac-d8246f775af0', vmId='1734fed5-fc97-4714-bb63-55e8f4eb6d34'}', device='ide', type='CONTROLLER', bootOrder='0', specParams='[]', address='{slot=0x01, bus=0x00, domain=0x0000, type=pci, function=0x1}', managed='false', plugged='true', readOnly='false', deviceAlias='ide', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_9d496570-3467-475b-ba9c-3111478eb9a7device_fb1efd5f-e403-44f3-ae46-9e817b19ee91device_953c69bc-7182-496a-8180-1a8dc3230464=VmDevice:{id='VmDeviceId:{deviceId='953c69bc-7182-496a-8180-1a8dc3230464', vmId='1734fed5-fc97-4714-bb63-55e8f4eb6d34'}', device='spicevmc', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=3}', managed='false', plugged='true', readOnly='false', deviceAlias='channel2', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}}, display=qxl, timeOffset=0, nice=0, maxMemSize=16384, maxMemSlots=16, bootMenuEnable=false, memSize=4096} 2017-10-01 16:35:05,620-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] FINISH, CreateVDSCommand, log id: 19dc4d4d 2017-10-01 16:35:05,622-04 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] FINISH, CreateVmVDSCommand, return: WaitForLaunch, log id: 39507290 2017-10-01 16:35:05,622-04 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] Lock freed to object 'EngineLock:{exclusiveLocks='[1734fed5-fc97-4714-bb63-55e8f4eb6d34=VM]', sharedLocks=''}' 2017-10-01 16:35:05,626-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] EVENT_ID: USER_STARTED_VM(153), Correlation ID: ef0a1d32-bc65-4a25-a600-995f2ad50edb, Job ID: 21061038-ed9c-408a-a715-a5e78042ce8a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM wiki was started by jas@eecs.yorku.ca-authz (Host: virt2). 2017-10-01 16:35:05,636-04 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] Running command: RunVmCommand internal: false. Entities affected : ID: c8682859-9e55-460a-9ef0-d03e4a85f50e Type: VMAction group RUN_VM with role type USER 2017-10-01 16:35:05,720-04 INFO [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] START, UpdateVmDynamicDataVDSCommand( UpdateVmDynamicDataVDSCommandParameters:{runAsync='true', hostId='null', vmId='c8682859-9e55-460a-9ef0-d03e4a85f50e', vmDynamic='org.ovirt.engine.core.common.businessentities.VmDynamic@4af022c3'}), log id: 6aedb56c 2017-10-01 16:35:05,721-04 INFO [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] FINISH, UpdateVmDynamicDataVDSCommand, log id: 6aedb56c 2017-10-01 16:35:05,723-04 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] START, CreateVmVDSCommand( CreateVmVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vmId='c8682859-9e55-460a-9ef0-d03e4a85f50e', vm='VM [dhcp1]'}), log id: 40752871 2017-10-01 16:35:05,725-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] START, CreateVDSCommand(HostName = virt4, CreateVmVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vmId='c8682859-9e55-460a-9ef0-d03e4a85f50e', vm='VM [dhcp1]'}), log id: 4f2640fc 2017-10-01 16:35:05,728-04 INFO [org.ovirt.engine.core.vdsbroker.builder.vminfo.VmInfoBuilderImpl] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] Bootable disk '674b7823-3f83-4a4c-a0fe-f0ba108ac192' set to index '0' 2017-10-01 16:35:05,740-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] VM {memGuaranteedSize=682, smpThreadsPerCore=1, cpuType=SandyBridge, vmId=c8682859-9e55-460a-9ef0-d03e4a85f50e, acpiEnable=true, vmType=kvm, smp=1, smpCoresPerSocket=1, emulatedMachine=pc-i440fx-rhel7.3.0, smartcardEnable=false, guestNumaNodes=[{memory=1024, cpus=0, nodeIndex=0}], transparentHugePages=true, displayNetwork=yorku, vmName=dhcp1, maxVCpus=16, kvmEnable=true, devices=[{address={bus=0x00, domain=0x0000, function=0x0, slot=0x02, type=pci}, type=video, specParams={heads=1, vram=32768}, device=qxl, deviceId=92ffae9b-28ed-429d-b1eb-048891f7a0b6}, {type=graphics, specParams={fileTransferEnable=true, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,ssmartcard,susbredir, copyPasteEnable=true}, device=spice, deviceId=71bf73f5-1c73-4e2a-8dea-0cb9402d5e4e}, {iface=ide, shared=false, path=, address={bus=1, controller=0, unit=0, type=drive, target=0}, readonly=true, index=2, type=disk, specParams={path=}, device=cdrom, deviceId=ce077615-ae89-410c-b620-65c5be1acd8f}, {discard=false, shared=false, address={bus=0x00, domain=0x0000, function=0x0, slot=0x07, type=pci}, imageID=674b7823-3f83-4a4c-a0fe-f0ba108ac192, format=raw, index=0, optional=false, type=disk, deviceId=674b7823-3f83-4a4c-a0fe-f0ba108ac192, domainID=c8653dba-63ec-4447-872c-0ffcc6af9c0f, propagateErrors=off, iface=virtio, readonly=false, bootOrder=1, poolID=aca61eac-ed8a-484b-bbee-3645148f6405, volumeID=a4b4f2f6-b2e2-4f04-86dd-7ef742733eec, specParams={}, device=disk}, {filter=vdsm-no-mac-spoofing, nicModel=pv, address={bus=0x00, domain=0x0000, function=0x0, slot=0x03, type=pci}, type=interface, specParams={inbound={}, outbound={}}, device=bridge, linkActive=true, deviceId=b72539b3-2623-4c5c-a265-b2d3f147f632, macAddr=00:1a:4a:32:3d:29, network=yorku}, {filter=vdsm-no-mac-spoofing, nicModel=pv, address={bus=0x00, domain=0x0000, function=0x0, slot=0x04, type=pci}, type=interface, specParams={inbound={}, outbound={}}, device=bridge, linkActive=true, deviceId=e8007eab-2331-4e78-a90e-8e111ba73dd9, macAddr=00:1a:4a:32:3d:2a, network=mrpriv}, {address={bus=0x00, domain=0x0000, function=0x2, slot=0x01, type=pci}, index=0, model=piix3-uhci, type=controller, specParams={}, device=usb, deviceId=e9fbad14-1c00-47cb-b5df-ca01b43b1560}, {address={bus=0x00, domain=0x0000, function=0x0, slot=0x05, type=pci}, index=0, model=virtio-scsi, type=controller, specParams={}, device=scsi, deviceId=d401de40-fe33-4a9d-9cb3-4a728e4fd068}, {address={bus=0x00, domain=0x0000, function=0x0, slot=0x06, type=pci}, type=controller, specParams={}, device=virtio-serial, deviceId=870bfcd1-ccdf-4b0a-9d81-288b2e6d5019}, {model=virtio, type=rng, specParams={source=urandom}, device=virtio, deviceId=0e6d53f8-c270-4a02-9ead-29768d4d5646}], custom={device_47d5dd52-0ce1-408e-bdb7-1fae094bb02ddevice_7b48c7c7-2bdb-4d5a-bda3-7c0a3b5b88b3=VmDevice:{id='VmDeviceId:{deviceId='7b48c7c7-2bdb-4d5a-bda3-7c0a3b5b88b3', vmId='c8682859-9e55-460a-9ef0-d03e4a85f50e'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=2}', managed='false', plugged='true', readOnly='false', deviceAlias='channel1', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_47d5dd52-0ce1-408e-bdb7-1fae094bb02ddevice_7b48c7c7-2bdb-4d5a-bda3-7c0a3b5b88b3device_da6c86d3-5d2b-4196-81d7-e89b374395d9=VmDevice:{id='VmDeviceId:{deviceId='da6c86d3-5d2b-4196-81d7-e89b374395d9', vmId='c8682859-9e55-460a-9ef0-d03e4a85f50e'}', device='spicevmc', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=3}', managed='false', plugged='true', readOnly='false', deviceAlias='channel2', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_47d5dd52-0ce1-408e-bdb7-1fae094bb02ddevice_7b48c7c7-2bdb-4d5a-bda3-7c0a3b5b88b3device_da6c86d3-5d2b-4196-81d7-e89b374395d9device_cb545b03-f085-40e4-8428-cd190e13b61a=VmDevice:{id='VmDeviceId:{deviceId='cb545b03-f085-40e4-8428-cd190e13b61a', vmId='c8682859-9e55-460a-9ef0-d03e4a85f50e'}', device='ide', type='CONTROLLER', bootOrder='0', specParams='[]', address='{slot=0x01, bus=0x00, domain=0x0000, type=pci, function=0x1}', managed='false', plugged='true', readOnly='false', deviceAlias='ide', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_47d5dd52-0ce1-408e-bdb7-1fae094bb02d=VmDevice:{id='VmDeviceId:{deviceId='47d5dd52-0ce1-408e-bdb7-1fae094bb02d', vmId='c8682859-9e55-460a-9ef0-d03e4a85f50e'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=1}', managed='false', plugged='true', readOnly='false', deviceAlias='channel0', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}}, display=qxl, timeOffset=0, nice=0, maxMemSize=4096, maxMemSlots=16, bootMenuEnable=false, memSize=1024} 2017-10-01 16:35:05,789-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] FINISH, CreateVDSCommand, log id: 4f2640fc 2017-10-01 16:35:05,792-04 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] FINISH, CreateVmVDSCommand, return: WaitForLaunch, log id: 40752871 2017-10-01 16:35:05,792-04 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] Lock freed to object 'EngineLock:{exclusiveLocks='[c8682859-9e55-460a-9ef0-d03e4a85f50e=VM]', sharedLocks=''}' 2017-10-01 16:35:05,796-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-50) [ef0a1d32-bc65-4a25-a600-995f2ad50edb] EVENT_ID: USER_STARTED_VM(153), Correlation ID: ef0a1d32-bc65-4a25-a600-995f2ad50edb, Job ID: 58769063-2c71-4d80-9d7d-b94db298bbc1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM dhcp1 was started by jas@eecs.yorku.ca-authz (Host: virt4). 2017-10-01 16:35:06,015-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmsStatisticsFetcher] (DefaultQuartzScheduler5) [11d49caa] Fetched 5 VMs from VDS 'a0ccfd11-3402-4eba-a119-4987561b738e' 2017-10-01 16:35:06,016-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler5) [11d49caa] VM 'a82b4cc3-6ef6-4a8e-97ef-1d4ea1b7e07c'(indigo) moved from 'WaitForLaunch' --> 'PoweringUp' 2017-10-01 16:35:06,016-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler5) [11d49caa] VM 'aa487207-7ff4-465a-9d9b-2a103d50dc77'(ftp1) moved from 'WaitForLaunch' --> 'PoweringUp' 2017-10-01 16:35:06,026-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVDSCommand] (DefaultQuartzScheduler5) [11d49caa] START, FullListVDSCommand(HostName = virt2, FullListVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vmIds='[9ea2821e-557f-45cb-9872-1e6c5aeaf246, a82b4cc3-6ef6-4a8e-97ef-1d4ea1b7e07c, aa487207-7ff4-465a-9d9b-2a103d50dc77, 1734fed5-fc97-4714-bb63-55e8f4eb6d34]'}), log id: 6ad0ebb2 2017-10-01 16:35:06,068-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVDSCommand] (DefaultQuartzScheduler5) [11d49caa] FINISH, FullListVDSCommand, return: [{acpiEnable=true, emulatedMachine=pc-i440fx-rhel7.3.0, vmId=aa487207-7ff4-465a-9d9b-2a103d50dc77, guestDiskMapping={}, transparentHugePages=true, timeOffset=0, cpuType=SandyBridge, smp=1, pauseCode=NOERR, guestNumaNodes=[Ljava.lang.Object;@741bc1ce, smartcardEnable=false, custom={device_21a77b7d-9169-4b85-a9da-862116b3348b=VmDevice:{id='VmDeviceId:{deviceId='21a77b7d-9169-4b85-a9da-862116b3348b', vmId='aa487207-7ff4-465a-9d9b-2a103d50dc77'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=1}', managed='false', plugged='true', readOnly='false', deviceAlias='channel0', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_21a77b7d-9169-4b85-a9da-862116b3348bdevice_ad098866-fd36-4876-850e-3530893b2d8e=VmDevice:{id='VmDeviceId:{deviceId='ad098866-fd36-4876-850e-3530893b2d8e', vmId='aa487207-7ff4-465a-9d9b-2a103d50dc77'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=2}', managed='false', plugged='true', readOnly='false', deviceAlias='channel1', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_21a77b7d-9169-4b85-a9da-862116b3348bdevice_ad098866-fd36-4876-850e-3530893b2d8edevice_b03fbbe1-3200-4876-a047-a33ff6e5c28f=VmDevice:{id='VmDeviceId:{deviceId='b03fbbe1-3200-4876-a047-a33ff6e5c28f', vmId='aa487207-7ff4-465a-9d9b-2a103d50dc77'}', device='spicevmc', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=3}', managed='false', plugged='true', readOnly='false', deviceAlias='channel2', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_21a77b7d-9169-4b85-a9da-862116b3348bdevice_ad098866-fd36-4876-850e-3530893b2d8edevice_b03fbbe1-3200-4876-a047-a33ff6e5c28fdevice_d00873ad-f4ea-4dec-aa73-99aa3b82a5ad=VmDevice:{id='VmDeviceId:{deviceId='d00873ad-f4ea-4dec-aa73-99aa3b82a5ad', vmId='aa487207-7ff4-465a-9d9b-2a103d50dc77'}', device='ide', type='CONTROLLER', bootOrder='0', specParams='[]', address='{slot=0x01, bus=0x00, domain=0x0000, type=pci, function=0x1}', managed='false', plugged='true', readOnly='false', deviceAlias='ide', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}}, vmType=kvm, memSize=2048, smpCoresPerSocket=1, vmName=ftp1, nice=0, status=Up, maxMemSize=8192, bootMenuEnable=false, pid=3164, smpThreadsPerCore=1, memGuaranteedSize=1365, kvmEnable=true, displayNetwork=yorku, devices=[Ljava.lang.Object;@72ef9034, display=qxl, maxVCpus=16, clientIp=, statusTime=4726789260, maxMemSlots=16}, {acpiEnable=true, emulatedMachine=pc-i440fx-rhel7.3.0, vmId=1734fed5-fc97-4714-bb63-55e8f4eb6d34, guestDiskMapping={}, transparentHugePages=true, timeOffset=0, cpuType=SandyBridge, smp=2, guestNumaNodes=[Ljava.lang.Object;@5094ea1c, smartcardEnable=false, custom={device_9d496570-3467-475b-ba9c-3111478eb9a7device_fb1efd5f-e403-44f3-ae46-9e817b19ee91=VmDevice:{id='VmDeviceId:{deviceId='fb1efd5f-e403-44f3-ae46-9e817b19ee91', vmId='1734fed5-fc97-4714-bb63-55e8f4eb6d34'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=2}', managed='false', plugged='true', readOnly='false', deviceAlias='channel1', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_9d496570-3467-475b-ba9c-3111478eb9a7device_fb1efd5f-e403-44f3-ae46-9e817b19ee91device_953c69bc-7182-496a-8180-1a8dc3230464device_9a7950a3-fb9b-4f82-adac-d8246f775af0=VmDevice:{id='VmDeviceId:{deviceId='9a7950a3-fb9b-4f82-adac-d8246f775af0', vmId='1734fed5-fc97-4714-bb63-55e8f4eb6d34'}', device='ide', type='CONTROLLER', bootOrder='0', specParams='[]', address='{slot=0x01, bus=0x00, domain=0x0000, type=pci, function=0x1}', managed='false', plugged='true', readOnly='false', deviceAlias='ide', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_9d496570-3467-475b-ba9c-3111478eb9a7=VmDevice:{id='VmDeviceId:{deviceId='9d496570-3467-475b-ba9c-3111478eb9a7', vmId='1734fed5-fc97-4714-bb63-55e8f4eb6d34'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=1}', managed='false', plugged='true', readOnly='false', deviceAlias='channel0', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_9d496570-3467-475b-ba9c-3111478eb9a7device_fb1efd5f-e403-44f3-ae46-9e817b19ee91device_953c69bc-7182-496a-8180-1a8dc3230464=VmDevice:{id='VmDeviceId:{deviceId='953c69bc-7182-496a-8180-1a8dc3230464', vmId='1734fed5-fc97-4714-bb63-55e8f4eb6d34'}', device='spicevmc', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=3}', managed='false', plugged='true', readOnly='false', deviceAlias='channel2', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}}, vmType=kvm, memSize=4096, smpCoresPerSocket=1, vmName=wiki, nice=0, status=WaitForLaunch, maxMemSize=16384, bootMenuEnable=false, pid=0, smpThreadsPerCore=1, memGuaranteedSize=4096, kvmEnable=true, displayNetwork=yorku, devices=[Ljava.lang.Object;@3df416c7, display=qxl, maxVCpus=16, clientIp=, statusTime=4726789260, maxMemSlots=16}, {acpiEnable=true, emulatedMachine=pc-i440fx-rhel7.3.0, vmId=a82b4cc3-6ef6-4a8e-97ef-1d4ea1b7e07c, guestDiskMapping={}, transparentHugePages=true, timeOffset=0, cpuType=SandyBridge, smp=4, pauseCode=NOERR, guestNumaNodes=[Ljava.lang.Object;@174ba506, smartcardEnable=false, custom={device_9c16a7ba-80f8-467e-8e1b-610ee15a6cbedevice_135814bd-ae90-432b-b5a5-9173e8b44ce6=VmDevice:{id='VmDeviceId:{deviceId='135814bd-ae90-432b-b5a5-9173e8b44ce6', vmId='a82b4cc3-6ef6-4a8e-97ef-1d4ea1b7e07c'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=2}', managed='false', plugged='true', readOnly='false', deviceAlias='channel1', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_9c16a7ba-80f8-467e-8e1b-610ee15a6cbedevice_135814bd-ae90-432b-b5a5-9173e8b44ce6device_f7655e22-cfd1-4388-ba00-35b7c29d2213=VmDevice:{id='VmDeviceId:{deviceId='f7655e22-cfd1-4388-ba00-35b7c29d2213', vmId='a82b4cc3-6ef6-4a8e-97ef-1d4ea1b7e07c'}', device='spicevmc', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=3}', managed='false', plugged='true', readOnly='false', deviceAlias='channel2', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_9c16a7ba-80f8-467e-8e1b-610ee15a6cbe=VmDevice:{id='VmDeviceId:{deviceId='9c16a7ba-80f8-467e-8e1b-610ee15a6cbe', vmId='a82b4cc3-6ef6-4a8e-97ef-1d4ea1b7e07c'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=1}', managed='false', plugged='true', readOnly='false', deviceAlias='channel0', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_9c16a7ba-80f8-467e-8e1b-610ee15a6cbedevice_135814bd-ae90-432b-b5a5-9173e8b44ce6device_f7655e22-cfd1-4388-ba00-35b7c29d2213device_bee4d6b1-a28d-4dd6-a6a3-3bab1d73a117=VmDevice:{id='VmDeviceId:{deviceId='bee4d6b1-a28d-4dd6-a6a3-3bab1d73a117', vmId='a82b4cc3-6ef6-4a8e-97ef-1d4ea1b7e07c'}', device='ide', type='CONTROLLER', bootOrder='0', specParams='[]', address='{slot=0x01, bus=0x00, domain=0x0000, type=pci, function=0x1}', managed='false', plugged='true', readOnly='false', deviceAlias='ide', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}}, vmType=kvm, memSize=16384, smpCoresPerSocket=1, vmName=indigo, nice=0, status=Up, maxMemSize=65536, bootMenuEnable=false, pid=3344, smpThreadsPerCore=1, memGuaranteedSize=16384, kvmEnable=true, displayNetwork=yorku, devices=[Ljava.lang.Object;@192bf9eb, display=qxl, maxVCpus=16, clientIp=, statusTime=4726789260, maxMemSlots=16}, {acpiEnable=true, emulatedMachine=pc-i440fx-rhel7.3.0, vmId=9ea2821e-557f-45cb-9872-1e6c5aeaf246, guestDiskMapping={}, transparentHugePages=true, timeOffset=0, cpuType=SandyBridge, smp=1, guestNumaNodes=[Ljava.lang.Object;@6e1834d8, smartcardEnable=false, custom={device_ede985f5-ebdf-4058-9bd6-85b32f6086d1device_210d38b2-63a2-4b1e-8d59-3ede51837fc8=VmDevice:{id='VmDeviceId:{deviceId='210d38b2-63a2-4b1e-8d59-3ede51837fc8', vmId='9ea2821e-557f-45cb-9872-1e6c5aeaf246'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=2}', managed='false', plugged='true', readOnly='false', deviceAlias='channel1', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_ede985f5-ebdf-4058-9bd6-85b32f6086d1device_210d38b2-63a2-4b1e-8d59-3ede51837fc8device_b674e26e-33c4-45f6-b6d6-a751958e7889device_64356cea-121e-49a6-b2e3-b61cf6ef581d=VmDevice:{id='VmDeviceId:{deviceId='64356cea-121e-49a6-b2e3-b61cf6ef581d', vmId='9ea2821e-557f-45cb-9872-1e6c5aeaf246'}', device='ide', type='CONTROLLER', bootOrder='0', specParams='[]', address='{slot=0x01, bus=0x00, domain=0x0000, type=pci, function=0x1}', managed='false', plugged='true', readOnly='false', deviceAlias='ide', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_ede985f5-ebdf-4058-9bd6-85b32f6086d1device_210d38b2-63a2-4b1e-8d59-3ede51837fc8device_b674e26e-33c4-45f6-b6d6-a751958e7889=VmDevice:{id='VmDeviceId:{deviceId='b674e26e-33c4-45f6-b6d6-a751958e7889', vmId='9ea2821e-557f-45cb-9872-1e6c5aeaf246'}', device='spicevmc', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=3}', managed='false', plugged='true', readOnly='false', deviceAlias='channel2', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_ede985f5-ebdf-4058-9bd6-85b32f6086d1=VmDevice:{id='VmDeviceId:{deviceId='ede985f5-ebdf-4058-9bd6-85b32f6086d1', vmId='9ea2821e-557f-45cb-9872-1e6c5aeaf246'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=1}', managed='false', plugged='true', readOnly='false', deviceAlias='channel0', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}}, vmType=kvm, memSize=1024, smpCoresPerSocket=1, vmName=rs, nice=0, status=WaitForLaunch, maxMemSize=4096, bootMenuEnable=false, pid=0, smpThreadsPerCore=1, memGuaranteedSize=682, kvmEnable=true, displayNetwork=yorku, devices=[Ljava.lang.Object;@734745f2, display=qxl, maxVCpus=16, clientIp=, statusTime=4726789260, maxMemSlots=16}], log id: 6ad0ebb2 2017-10-01 16:35:06,071-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler5) [11d49caa] Received a spice Device without an address when processing VM aa487207-7ff4-465a-9d9b-2a103d50dc77 devices, skipping device: {device=spice, specParams={fileTransferEnable=true, displayNetwork=yorku, copyPasteEnable=true, displayIp=130.63.94.35, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,ssmartcard,susbredir}, type=graphics, deviceId=00000024-0024-0024-0024-0000000001c3, tlsPort=5901} 2017-10-01 16:35:06,072-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler5) [11d49caa] Received a memballoon Device without an address when processing VM 1734fed5-fc97-4714-bb63-55e8f4eb6d34 devices, skipping device: {device=memballoon, specParams={model=none}, type=balloon} 2017-10-01 16:35:06,072-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler5) [11d49caa] Received a virtio Device without an address when processing VM 1734fed5-fc97-4714-bb63-55e8f4eb6d34 devices, skipping device: {device=virtio, specParams={source=urandom}, model=virtio, type=rng, deviceId=ae6c7e6e-b9d4-4b03-8593-1f0e6adbf707} 2017-10-01 16:35:06,072-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler5) [11d49caa] Received a spice Device without an address when processing VM 1734fed5-fc97-4714-bb63-55e8f4eb6d34 devices, skipping device: {device=spice, specParams={fileTransferEnable=true, displayNetwork=yorku, copyPasteEnable=true, displayIp=130.63.94.35, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,ssmartcard,susbredir}, type=graphics, deviceId=0000002d-002d-002d-002d-0000000001ac} 2017-10-01 16:35:06,073-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler5) [11d49caa] Received a memballoon Device without an address when processing VM a82b4cc3-6ef6-4a8e-97ef-1d4ea1b7e07c devices, skipping device: {device=memballoon, specParams={model=none}, type=balloon, alias=balloon0} 2017-10-01 16:35:06,073-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler5) [11d49caa] Received a spice Device without an address when processing VM a82b4cc3-6ef6-4a8e-97ef-1d4ea1b7e07c devices, skipping device: {device=spice, specParams={fileTransferEnable=true, displayNetwork=yorku, copyPasteEnable=true, displayIp=130.63.94.35, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,ssmartcard,susbredir}, type=graphics, deviceId=0000002a-002a-002a-002a-000000000166, tlsPort=5902} 2017-10-01 16:35:06,074-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler5) [11d49caa] Received a memballoon Device without an address when processing VM 9ea2821e-557f-45cb-9872-1e6c5aeaf246 devices, skipping device: {device=memballoon, specParams={model=none}, type=balloon} 2017-10-01 16:35:06,074-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler5) [11d49caa] Received a virtio Device without an address when processing VM 9ea2821e-557f-45cb-9872-1e6c5aeaf246 devices, skipping device: {device=virtio, specParams={source=urandom}, model=virtio, type=rng, deviceId=0915ea55-b389-4808-883e-cad240bfa869} 2017-10-01 16:35:06,074-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler5) [11d49caa] Received a spice Device without an address when processing VM 9ea2821e-557f-45cb-9872-1e6c5aeaf246 devices, skipping device: {device=spice, specParams={fileTransferEnable=true, displayNetwork=yorku, copyPasteEnable=true, displayIp=130.63.94.35, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,ssmartcard,susbredir}, type=graphics, deviceId=58c8cadc-6a62-488c-85d2-dc06c41907f3} 2017-10-01 16:35:10,291-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmsStatisticsFetcher] (DefaultQuartzScheduler7) [39055b3d] Fetched 8 VMs from VDS '5ae29ed2-9c0f-4754-8df3-b43d14726846' 2017-10-01 16:35:10,292-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler7) [39055b3d] VM '3e5a5728-8940-4fe7-b4af-11fb8e12f116'(www) moved from 'WaitForLaunch' --> 'PoweringUp' 2017-10-01 16:35:10,292-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler7) [39055b3d] VM 'f87ec08d-0fdf-42ba-8aac-c66c404d7aae'(webapp) moved from 'WaitForLaunch' --> 'PoweringUp' 2017-10-01 16:35:10,292-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler7) [39055b3d] VM '911d6883-607c-464e-b46b-8e6141bdd2de'(hdb) moved from 'WaitForLaunch' --> 'PoweringUp' 2017-10-01 16:35:10,292-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler7) [39055b3d] VM 'bfe9a524-2238-4bab-a91f-5dc6ed270896'(pcmode) moved from 'WaitForLaunch' --> 'PoweringUp' 2017-10-01 16:35:10,292-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler7) [39055b3d] VM 'c8682859-9e55-460a-9ef0-d03e4a85f50e'(dhcp1) moved from 'WaitForLaunch' --> 'PoweringUp' 2017-10-01 16:35:10,292-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler7) [39055b3d] VM '64734fca-4d6c-443c-82d1-abf7cf106601'(webdav) moved from 'WaitForLaunch' --> 'PoweringUp' 2017-10-01 16:35:10,321-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVDSCommand] (DefaultQuartzScheduler7) [39055b3d] START, FullListVDSCommand(HostName = virt4, FullListVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vmIds='[911d6883-607c-464e-b46b-8e6141bdd2de, ae56c31c-71f7-433a-b198-a1c2c846325e, bfe9a524-2238-4bab-a91f-5dc6ed270896, c8682859-9e55-460a-9ef0-d03e4a85f50e, f87ec08d-0fdf-42ba-8aac-c66c404d7aae, 3e5a5728-8940-4fe7-b4af-11fb8e12f116, 64734fca-4d6c-443c-82d1-abf7cf106601]'}), log id: 5d344786 2017-10-01 16:35:10,387-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVDSCommand] (DefaultQuartzScheduler7) [39055b3d] FINISH, FullListVDSCommand, return: [{acpiEnable=true, emulatedMachine=pc-i440fx-rhel7.3.0, vmId=911d6883-607c-464e-b46b-8e6141bdd2de, guestDiskMapping={}, transparentHugePages=true, timeOffset=0, cpuType=SandyBridge, smp=1, pauseCode=NOERR, guestNumaNodes=[Ljava.lang.Object;@272ed8fa, smartcardEnable=false, custom={device_200d4fa2-8a9a-43e5-b769-3420de44e63d=VmDevice:{id='VmDeviceId:{deviceId='200d4fa2-8a9a-43e5-b769-3420de44e63d', vmId='911d6883-607c-464e-b46b-8e6141bdd2de'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=1}', managed='false', plugged='true', readOnly='false', deviceAlias='channel0', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_200d4fa2-8a9a-43e5-b769-3420de44e63ddevice_ef6ba1cf-fdfb-4ec5-9b48-48d5e5ce8ad9=VmDevice:{id='VmDeviceId:{deviceId='ef6ba1cf-fdfb-4ec5-9b48-48d5e5ce8ad9', vmId='911d6883-607c-464e-b46b-8e6141bdd2de'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=2}', managed='false', plugged='true', readOnly='false', deviceAlias='channel1', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_200d4fa2-8a9a-43e5-b769-3420de44e63ddevice_ef6ba1cf-fdfb-4ec5-9b48-48d5e5ce8ad9device_10e4eeaa-a048-45db-baad-94fe84486584device_cadec10e-b41e-4470-a3b0-6bc21d303ba3=VmDevice:{id='VmDeviceId:{deviceId='cadec10e-b41e-4470-a3b0-6bc21d303ba3', vmId='911d6883-607c-464e-b46b-8e6141bdd2de'}', device='ide', type='CONTROLLER', bootOrder='0', specParams='[]', address='{slot=0x01, bus=0x00, domain=0x0000, type=pci, function=0x1}', managed='false', plugged='true', readOnly='false', deviceAlias='ide', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_200d4fa2-8a9a-43e5-b769-3420de44e63ddevice_ef6ba1cf-fdfb-4ec5-9b48-48d5e5ce8ad9device_10e4eeaa-a048-45db-baad-94fe84486584=VmDevice:{id='VmDeviceId:{deviceId='10e4eeaa-a048-45db-baad-94fe84486584', vmId='911d6883-607c-464e-b46b-8e6141bdd2de'}', device='spicevmc', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=3}', managed='false', plugged='true', readOnly='false', deviceAlias='channel2', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}}, vmType=kvm, memSize=1024, smpCoresPerSocket=1, vmName=hdb, nice=0, status=Up, maxMemSize=4096, bootMenuEnable=false, pid=108010, smpThreadsPerCore=1, memGuaranteedSize=682, kvmEnable=true, displayNetwork=yorku, devices=[Ljava.lang.Object;@3e227ae4, display=qxl, maxVCpus=16, clientIp=, statusTime=5257415000, maxMemSlots=16}, {acpiEnable=true, emulatedMachine=pc-i440fx-rhel7.3.0, vmId=f87ec08d-0fdf-42ba-8aac-c66c404d7aae, guestDiskMapping={}, transparentHugePages=true, timeOffset=0, cpuType=SandyBridge, smp=4, pauseCode=NOERR, guestNumaNodes=[Ljava.lang.Object;@1bc1d9f0, smartcardEnable=false, custom={device_b626f5c1-2022-4b58-a584-0c3b5b743a6b=VmDevice:{id='VmDeviceId:{deviceId='b626f5c1-2022-4b58-a584-0c3b5b743a6b', vmId='f87ec08d-0fdf-42ba-8aac-c66c404d7aae'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=1}', managed='false', plugged='true', readOnly='false', deviceAlias='channel0', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_b626f5c1-2022-4b58-a584-0c3b5b743a6bdevice_3c2d8356-f53a-4d8b-b8f9-3702b2619903device_5e4f0277-ca58-4f3f-87ad-4c836454484edevice_cd06efc2-9ec3-4642-858c-fd2d309e830d=VmDevice:{id='VmDeviceId:{deviceId='cd06efc2-9ec3-4642-858c-fd2d309e830d', vmId='f87ec08d-0fdf-42ba-8aac-c66c404d7aae'}', device='ide', type='CONTROLLER', bootOrder='0', specParams='[]', address='{slot=0x01, bus=0x00, domain=0x0000, type=pci, function=0x1}', managed='false', plugged='true', readOnly='false', deviceAlias='ide', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_b626f5c1-2022-4b58-a584-0c3b5b743a6bdevice_3c2d8356-f53a-4d8b-b8f9-3702b2619903device_5e4f0277-ca58-4f3f-87ad-4c836454484e=VmDevice:{id='VmDeviceId:{deviceId='5e4f0277-ca58-4f3f-87ad-4c836454484e', vmId='f87ec08d-0fdf-42ba-8aac-c66c404d7aae'}', device='spicevmc', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=3}', managed='false', plugged='true', readOnly='false', deviceAlias='channel2', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_b626f5c1-2022-4b58-a584-0c3b5b743a6bdevice_3c2d8356-f53a-4d8b-b8f9-3702b2619903=VmDevice:{id='VmDeviceId:{deviceId='3c2d8356-f53a-4d8b-b8f9-3702b2619903', vmId='f87ec08d-0fdf-42ba-8aac-c66c404d7aae'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=2}', managed='false', plugged='true', readOnly='false', deviceAlias='channel1', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}}, vmType=kvm, memSize=4096, smpCoresPerSocket=1, vmName=webapp, nice=0, status=Up, maxMemSize=16384, bootMenuEnable=false, pid=107659, smpThreadsPerCore=1, memGuaranteedSize=4096, kvmEnable=true, displayNetwork=yorku, devices=[Ljava.lang.Object;@6436b579, display=qxl, maxVCpus=16, clientIp=, statusTime=5257415000, maxMemSlots=16}, {acpiEnable=true, emulatedMachine=pc-i440fx-rhel7.3.0, vmId=c8682859-9e55-460a-9ef0-d03e4a85f50e, guestDiskMapping={}, transparentHugePages=true, timeOffset=0, cpuType=SandyBridge, smp=1, pauseCode=NOERR, guestNumaNodes=[Ljava.lang.Object;@358e580d, smartcardEnable=false, custom={device_47d5dd52-0ce1-408e-bdb7-1fae094bb02ddevice_7b48c7c7-2bdb-4d5a-bda3-7c0a3b5b88b3device_da6c86d3-5d2b-4196-81d7-e89b374395d9device_cb545b03-f085-40e4-8428-cd190e13b61a=VmDevice:{id='VmDeviceId:{deviceId='cb545b03-f085-40e4-8428-cd190e13b61a', vmId='c8682859-9e55-460a-9ef0-d03e4a85f50e'}', device='ide', type='CONTROLLER', bootOrder='0', specParams='[]', address='{slot=0x01, bus=0x00, domain=0x0000, type=pci, function=0x1}', managed='false', plugged='true', readOnly='false', deviceAlias='ide', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_47d5dd52-0ce1-408e-bdb7-1fae094bb02ddevice_7b48c7c7-2bdb-4d5a-bda3-7c0a3b5b88b3device_da6c86d3-5d2b-4196-81d7-e89b374395d9=VmDevice:{id='VmDeviceId:{deviceId='da6c86d3-5d2b-4196-81d7-e89b374395d9', vmId='c8682859-9e55-460a-9ef0-d03e4a85f50e'}', device='spicevmc', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=3}', managed='false', plugged='true', readOnly='false', deviceAlias='channel2', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_47d5dd52-0ce1-408e-bdb7-1fae094bb02ddevice_7b48c7c7-2bdb-4d5a-bda3-7c0a3b5b88b3=VmDevice:{id='VmDeviceId:{deviceId='7b48c7c7-2bdb-4d5a-bda3-7c0a3b5b88b3', vmId='c8682859-9e55-460a-9ef0-d03e4a85f50e'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=2}', managed='false', plugged='true', readOnly='false', deviceAlias='channel1', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_47d5dd52-0ce1-408e-bdb7-1fae094bb02d=VmDevice:{id='VmDeviceId:{deviceId='47d5dd52-0ce1-408e-bdb7-1fae094bb02d', vmId='c8682859-9e55-460a-9ef0-d03e4a85f50e'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=1}', managed='false', plugged='true', readOnly='false', deviceAlias='channel0', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}}, vmType=kvm, memSize=1024, smpCoresPerSocket=1, vmName=dhcp1, nice=0, status=Up, maxMemSize=4096, bootMenuEnable=false, pid=108561, smpThreadsPerCore=1, memGuaranteedSize=682, kvmEnable=true, displayNetwork=yorku, devices=[Ljava.lang.Object;@1856f102, display=qxl, maxVCpus=16, clientIp=, statusTime=5257415000, maxMemSlots=16}, {acpiEnable=true, emulatedMachine=pc-i440fx-rhel7.3.0, vmId=bfe9a524-2238-4bab-a91f-5dc6ed270896, guestDiskMapping={}, transparentHugePages=true, timeOffset=0, cpuType=SandyBridge, smp=1, pauseCode=NOERR, guestNumaNodes=[Ljava.lang.Object;@dcda5a1, smartcardEnable=false, custom={device_824b2add-bc03-46e1-9c2e-b9a6db1446ffdevice_aeaf7e01-9c68-4e9b-885f-bff2a751bd2edevice_99751d4e-6998-4ece-ba2d-3938d4a037d6device_536ebb04-ccad-43a0-8df5-c58ff732ef52=VmDevice:{id='VmDeviceId:{deviceId='536ebb04-ccad-43a0-8df5-c58ff732ef52', vmId='bfe9a524-2238-4bab-a91f-5dc6ed270896'}', device='ide', type='CONTROLLER', bootOrder='0', specParams='[]', address='{slot=0x01, bus=0x00, domain=0x0000, type=pci, function=0x1}', managed='false', plugged='true', readOnly='false', deviceAlias='ide', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_824b2add-bc03-46e1-9c2e-b9a6db1446ff=VmDevice:{id='VmDeviceId:{deviceId='824b2add-bc03-46e1-9c2e-b9a6db1446ff', vmId='bfe9a524-2238-4bab-a91f-5dc6ed270896'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=1}', managed='false', plugged='true', readOnly='false', deviceAlias='channel0', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_824b2add-bc03-46e1-9c2e-b9a6db1446ffdevice_aeaf7e01-9c68-4e9b-885f-bff2a751bd2e=VmDevice:{id='VmDeviceId:{deviceId='aeaf7e01-9c68-4e9b-885f-bff2a751bd2e', vmId='bfe9a524-2238-4bab-a91f-5dc6ed270896'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=2}', managed='false', plugged='true', readOnly='false', deviceAlias='channel1', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_824b2add-bc03-46e1-9c2e-b9a6db1446ffdevice_aeaf7e01-9c68-4e9b-885f-bff2a751bd2edevice_99751d4e-6998-4ece-ba2d-3938d4a037d6=VmDevice:{id='VmDeviceId:{deviceId='99751d4e-6998-4ece-ba2d-3938d4a037d6', vmId='bfe9a524-2238-4bab-a91f-5dc6ed270896'}', device='spicevmc', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=3}', managed='false', plugged='true', readOnly='false', deviceAlias='channel2', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}}, vmType=kvm, memSize=4096, smpCoresPerSocket=1, vmName=pcmode, nice=0, status=Up, maxMemSize=16384, bootMenuEnable=false, pid=107486, smpThreadsPerCore=1, memGuaranteedSize=4096, kvmEnable=true, displayNetwork=yorku, devices=[Ljava.lang.Object;@5c81541c, display=qxl, maxVCpus=16, clientIp=, statusTime=5257415000, maxMemSlots=16}, {acpiEnable=true, emulatedMachine=pc-i440fx-rhel7.3.0, vmId=64734fca-4d6c-443c-82d1-abf7cf106601, guestDiskMapping={}, transparentHugePages=true, timeOffset=0, cpuType=SandyBridge, smp=1, pauseCode=NOERR, guestNumaNodes=[Ljava.lang.Object;@1b8821b1, smartcardEnable=false, custom={device_729e6b0e-f43d-48a8-8407-2b824a5bf38cdevice_94e2e5e8-291a-4354-9c28-82e9c5e921af=VmDevice:{id='VmDeviceId:{deviceId='94e2e5e8-291a-4354-9c28-82e9c5e921af', vmId='64734fca-4d6c-443c-82d1-abf7cf106601'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=1}', managed='false', plugged='true', readOnly='false', deviceAlias='channel0', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_729e6b0e-f43d-48a8-8407-2b824a5bf38cdevice_94e2e5e8-291a-4354-9c28-82e9c5e921afdevice_87531847-0176-468a-b1e3-f2fc0c53ced6=VmDevice:{id='VmDeviceId:{deviceId='87531847-0176-468a-b1e3-f2fc0c53ced6', vmId='64734fca-4d6c-443c-82d1-abf7cf106601'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=2}', managed='false', plugged='true', readOnly='false', deviceAlias='channel1', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_729e6b0e-f43d-48a8-8407-2b824a5bf38c=VmDevice:{id='VmDeviceId:{deviceId='729e6b0e-f43d-48a8-8407-2b824a5bf38c', vmId='64734fca-4d6c-443c-82d1-abf7cf106601'}', device='ide', type='CONTROLLER', bootOrder='0', specParams='[]', address='{slot=0x01, bus=0x00, domain=0x0000, type=pci, function=0x1}', managed='false', plugged='true', readOnly='false', deviceAlias='ide', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_729e6b0e-f43d-48a8-8407-2b824a5bf38cdevice_94e2e5e8-291a-4354-9c28-82e9c5e921afdevice_87531847-0176-468a-b1e3-f2fc0c53ced6device_4e280ea3-c260-4154-b63a-0779beaff05f=VmDevice:{id='VmDeviceId:{deviceId='4e280ea3-c260-4154-b63a-0779beaff05f', vmId='64734fca-4d6c-443c-82d1-abf7cf106601'}', device='spicevmc', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=3}', managed='false', plugged='true', readOnly='false', deviceAlias='channel2', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}}, vmType=kvm, memSize=1024, smpCoresPerSocket=1, vmName=webdav, nice=0, status=Up, maxMemSize=4096, bootMenuEnable=false, pid=108183, smpThreadsPerCore=1, memGuaranteedSize=682, kvmEnable=true, displayNetwork=yorku, devices=[Ljava.lang.Object;@34f91d23, display=qxl, maxVCpus=16, clientIp=, statusTime=5257415000, maxMemSlots=16}, {acpiEnable=true, emulatedMachine=pc-i440fx-rhel7.3.0, vmId=3e5a5728-8940-4fe7-b4af-11fb8e12f116, guestDiskMapping={}, transparentHugePages=true, timeOffset=0, cpuType=SandyBridge, smp=4, pauseCode=NOERR, guestNumaNodes=[Ljava.lang.Object;@54f946b0, smartcardEnable=false, custom={device_5f45d2ad-18c0-4c9e-b2db-191071e278f5device_f996b11d-e7a3-43fb-9600-a5b5df2c7b6b=VmDevice:{id='VmDeviceId:{deviceId='f996b11d-e7a3-43fb-9600-a5b5df2c7b6b', vmId='3e5a5728-8940-4fe7-b4af-11fb8e12f116'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=2}', managed='false', plugged='true', readOnly='false', deviceAlias='channel1', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_5f45d2ad-18c0-4c9e-b2db-191071e278f5=VmDevice:{id='VmDeviceId:{deviceId='5f45d2ad-18c0-4c9e-b2db-191071e278f5', vmId='3e5a5728-8940-4fe7-b4af-11fb8e12f116'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=1}', managed='false', plugged='true', readOnly='false', deviceAlias='channel0', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_5f45d2ad-18c0-4c9e-b2db-191071e278f5device_f996b11d-e7a3-43fb-9600-a5b5df2c7b6bdevice_ad99aae2-6ca2-4694-ad3d-634037642322=VmDevice:{id='VmDeviceId:{deviceId='ad99aae2-6ca2-4694-ad3d-634037642322', vmId='3e5a5728-8940-4fe7-b4af-11fb8e12f116'}', device='spicevmc', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=3}', managed='false', plugged='true', readOnly='false', deviceAlias='channel2', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_5f45d2ad-18c0-4c9e-b2db-191071e278f5device_f996b11d-e7a3-43fb-9600-a5b5df2c7b6bdevice_ad99aae2-6ca2-4694-ad3d-634037642322device_b865730b-1acd-47ad-9a6a-28c09906ad75=VmDevice:{id='VmDeviceId:{deviceId='b865730b-1acd-47ad-9a6a-28c09906ad75', vmId='3e5a5728-8940-4fe7-b4af-11fb8e12f116'}', device='ide', type='CONTROLLER', bootOrder='0', specParams='[]', address='{slot=0x01, bus=0x00, domain=0x0000, type=pci, function=0x1}', managed='false', plugged='true', readOnly='false', deviceAlias='ide', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}}, vmType=kvm, memSize=4096, smpCoresPerSocket=1, vmName=www, nice=0, status=Up, maxMemSize=16384, bootMenuEnable=false, pid=107837, smpThreadsPerCore=1, memGuaranteedSize=4096, kvmEnable=true, displayNetwork=yorku, devices=[Ljava.lang.Object;@28ed987, display=qxl, maxVCpus=16, clientIp=, statusTime=5257415000, maxMemSlots=16}, {acpiEnable=true, emulatedMachine=pc-i440fx-rhel7.3.0, vmId=ae56c31c-71f7-433a-b198-a1c2c846325e, guestDiskMapping={}, transparentHugePages=true, timeOffset=0, cpuType=SandyBridge, smp=2, guestNumaNodes=[Ljava.lang.Object;@5851f5ce, smartcardEnable=false, custom={device_80a61fdb-b84a-4c66-bbcc-4a3a257b583f=VmDevice:{id='VmDeviceId:{deviceId='80a61fdb-b84a-4c66-bbcc-4a3a257b583f', vmId='ae56c31c-71f7-433a-b198-a1c2c846325e'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=1}', managed='false', plugged='true', readOnly='false', deviceAlias='channel0', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_80a61fdb-b84a-4c66-bbcc-4a3a257b583fdevice_5f697df1-cc6b-402f-8b25-ad143b5ee816device_f660a99d-ec1f-4a44-8f99-ae21606bfd34device_62c8af1e-b59b-4bd8-99dd-40948e53b6df=VmDevice:{id='VmDeviceId:{deviceId='62c8af1e-b59b-4bd8-99dd-40948e53b6df', vmId='ae56c31c-71f7-433a-b198-a1c2c846325e'}', device='ide', type='CONTROLLER', bootOrder='0', specParams='[]', address='{slot=0x01, bus=0x00, domain=0x0000, type=pci, function=0x1}', managed='false', plugged='true', readOnly='false', deviceAlias='ide', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_80a61fdb-b84a-4c66-bbcc-4a3a257b583fdevice_5f697df1-cc6b-402f-8b25-ad143b5ee816device_f660a99d-ec1f-4a44-8f99-ae21606bfd34=VmDevice:{id='VmDeviceId:{deviceId='f660a99d-ec1f-4a44-8f99-ae21606bfd34', vmId='ae56c31c-71f7-433a-b198-a1c2c846325e'}', device='spicevmc', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=3}', managed='false', plugged='true', readOnly='false', deviceAlias='channel2', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_80a61fdb-b84a-4c66-bbcc-4a3a257b583fdevice_5f697df1-cc6b-402f-8b25-ad143b5ee816=VmDevice:{id='VmDeviceId:{deviceId='5f697df1-cc6b-402f-8b25-ad143b5ee816', vmId='ae56c31c-71f7-433a-b198-a1c2c846325e'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=2}', managed='false', plugged='true', readOnly='false', deviceAlias='channel1', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}}, vmType=kvm, memSize=4096, smpCoresPerSocket=1, vmName=forum, nice=0, status=WaitForLaunch, maxMemSize=16384, bootMenuEnable=false, pid=0, smpThreadsPerCore=1, memGuaranteedSize=4096, kvmEnable=true, displayNetwork=yorku, devices=[Ljava.lang.Object;@243debe2, display=qxl, maxVCpus=16, clientIp=, statusTime=5257415000, maxMemSlots=16}], log id: 5d344786 2017-10-01 16:35:10,390-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler7) [39055b3d] Received a memballoon Device without an address when processing VM 911d6883-607c-464e-b46b-8e6141bdd2de devices, skipping device: {device=memballoon, specParams={model=none}, type=balloon, alias=balloon0} 2017-10-01 16:35:10,390-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler7) [39055b3d] Received a spice Device without an address when processing VM 911d6883-607c-464e-b46b-8e6141bdd2de devices, skipping device: {device=spice, specParams={fileTransferEnable=true, displayNetwork=yorku, copyPasteEnable=true, displayIp=130.63.94.37, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,ssmartcard,susbredir}, type=graphics, deviceId=62ba37d1-71cb-4c67-ac3c-bab7f7cf57b2, tlsPort=5904} 2017-10-01 16:35:10,391-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler7) [39055b3d] Received a memballoon Device without an address when processing VM f87ec08d-0fdf-42ba-8aac-c66c404d7aae devices, skipping device: {device=memballoon, specParams={model=none}, type=balloon, alias=balloon0} 2017-10-01 16:35:10,391-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler7) [39055b3d] Received a spice Device without an address when processing VM f87ec08d-0fdf-42ba-8aac-c66c404d7aae devices, skipping device: {device=spice, specParams={fileTransferEnable=true, displayNetwork=yorku, copyPasteEnable=true, displayIp=130.63.94.37, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,ssmartcard,susbredir}, type=graphics, deviceId=0000001e-001e-001e-001e-0000000002bc, tlsPort=5902} 2017-10-01 16:35:10,392-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler7) [39055b3d] Received a memballoon Device without an address when processing VM c8682859-9e55-460a-9ef0-d03e4a85f50e devices, skipping device: {device=memballoon, specParams={model=none}, type=balloon, alias=balloon0} 2017-10-01 16:35:10,392-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler7) [39055b3d] Received a spice Device without an address when processing VM c8682859-9e55-460a-9ef0-d03e4a85f50e devices, skipping device: {device=spice, specParams={fileTransferEnable=true, displayNetwork=yorku, copyPasteEnable=true, displayIp=130.63.94.37, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,ssmartcard,susbredir}, type=graphics, deviceId=71bf73f5-1c73-4e2a-8dea-0cb9402d5e4e, tlsPort=5907} 2017-10-01 16:35:10,393-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler7) [39055b3d] Received a memballoon Device without an address when processing VM bfe9a524-2238-4bab-a91f-5dc6ed270896 devices, skipping device: {device=memballoon, specParams={model=none}, type=balloon, alias=balloon0} 2017-10-01 16:35:10,393-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler7) [39055b3d] Received a spice Device without an address when processing VM bfe9a524-2238-4bab-a91f-5dc6ed270896 devices, skipping device: {device=spice, specParams={fileTransferEnable=true, displayNetwork=yorku, copyPasteEnable=true, displayIp=130.63.94.37, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,ssmartcard,susbredir}, type=graphics, deviceId=ca0f5b5f-d51f-4838-8042-a1e14722e179, tlsPort=5901} 2017-10-01 16:35:10,393-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler7) [39055b3d] Received a spice Device without an address when processing VM 64734fca-4d6c-443c-82d1-abf7cf106601 devices, skipping device: {device=spice, specParams={fileTransferEnable=true, displayNetwork=yorku, copyPasteEnable=true, displayIp=130.63.94.37, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,ssmartcard,susbredir}, type=graphics, deviceId=00000021-0021-0021-0021-00000000022e, tlsPort=5905} 2017-10-01 16:35:10,394-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler7) [39055b3d] Received a memballoon Device without an address when processing VM 3e5a5728-8940-4fe7-b4af-11fb8e12f116 devices, skipping device: {device=memballoon, specParams={model=none}, type=balloon, alias=balloon0} 2017-10-01 16:35:10,394-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler7) [39055b3d] Received a spice Device without an address when processing VM 3e5a5728-8940-4fe7-b4af-11fb8e12f116 devices, skipping device: {device=spice, specParams={fileTransferEnable=true, displayNetwork=yorku, copyPasteEnable=true, displayIp=130.63.94.37, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,ssmartcard,susbredir}, type=graphics, deviceId=0000002c-002c-002c-002c-000000000240, tlsPort=5903} 2017-10-01 16:35:10,395-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler7) [39055b3d] Received a memballoon Device without an address when processing VM ae56c31c-71f7-433a-b198-a1c2c846325e devices, skipping device: {device=memballoon, specParams={model=none}, type=balloon} 2017-10-01 16:35:10,395-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler7) [39055b3d] Received a virtio Device without an address when processing VM ae56c31c-71f7-433a-b198-a1c2c846325e devices, skipping device: {device=virtio, specParams={source=urandom}, model=virtio, type=rng, deviceId=9ae72c8c-e2a9-4810-83c6-1549ffa10ad8} 2017-10-01 16:35:10,395-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler7) [39055b3d] Received a spice Device without an address when processing VM ae56c31c-71f7-433a-b198-a1c2c846325e devices, skipping device: {device=spice, specParams={fileTransferEnable=true, displayNetwork=yorku, copyPasteEnable=true, displayIp=130.63.94.37, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,ssmartcard,susbredir}, type=graphics, deviceId=00000030-0030-0030-0030-0000000001cd} 2017-10-01 16:35:11,140-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmsStatisticsFetcher] (DefaultQuartzScheduler8) [1f94017c] Fetched 4 VMs from VDS '45f8b331-842e-48e7-9df8-56adddb93836' 2017-10-01 16:35:11,140-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler8) [1f94017c] VM '1c7e966f-29e8-4788-b64c-86e52f415061'(udb) moved from 'WaitForLaunch' --> 'PoweringUp' 2017-10-01 16:35:11,141-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler8) [1f94017c] VM 'ae65a104-ed3f-40e0-ba92-5708cf338918'(vpn2) moved from 'WaitForLaunch' --> 'PoweringUp' 2017-10-01 16:35:11,169-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVDSCommand] (DefaultQuartzScheduler8) [1f94017c] START, FullListVDSCommand(HostName = virt1, FullListVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vmIds='[ae65a104-ed3f-40e0-ba92-5708cf338918, c64d80f9-a3bd-44f3-89c4-f1b9b509cdde, 1c7e966f-29e8-4788-b64c-86e52f415061]'}), log id: 7b565bee 2017-10-01 16:35:11,201-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVDSCommand] (DefaultQuartzScheduler8) [1f94017c] FINISH, FullListVDSCommand, return: [{acpiEnable=true, emulatedMachine=pc-i440fx-rhel7.3.0, vmId=c64d80f9-a3bd-44f3-89c4-f1b9b509cdde, guestDiskMapping={}, transparentHugePages=true, timeOffset=0, cpuType=SandyBridge, smp=6, guestNumaNodes=[Ljava.lang.Object;@6714a60d, smartcardEnable=false, custom={device_16dad27f-1884-4f32-8736-6bbaf8cd2fbddevice_d9f2931e-e8e0-4b83-8534-9c35daabff71device_28c1e908-e275-411a-bbeb-2920a3032e74=VmDevice:{id='VmDeviceId:{deviceId='28c1e908-e275-411a-bbeb-2920a3032e74', vmId='c64d80f9-a3bd-44f3-89c4-f1b9b509cdde'}', device='spicevmc', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=3}', managed='false', plugged='true', readOnly='false', deviceAlias='channel2', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_16dad27f-1884-4f32-8736-6bbaf8cd2fbddevice_d9f2931e-e8e0-4b83-8534-9c35daabff71=VmDevice:{id='VmDeviceId:{deviceId='d9f2931e-e8e0-4b83-8534-9c35daabff71', vmId='c64d80f9-a3bd-44f3-89c4-f1b9b509cdde'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=2}', managed='false', plugged='true', readOnly='false', deviceAlias='channel1', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_16dad27f-1884-4f32-8736-6bbaf8cd2fbddevice_d9f2931e-e8e0-4b83-8534-9c35daabff71device_28c1e908-e275-411a-bbeb-2920a3032e74device_44b628f3-6129-417b-a46d-de82d1b20d76=VmDevice:{id='VmDeviceId:{deviceId='44b628f3-6129-417b-a46d-de82d1b20d76', vmId='c64d80f9-a3bd-44f3-89c4-f1b9b509cdde'}', device='ide', type='CONTROLLER', bootOrder='0', specParams='[]', address='{slot=0x01, bus=0x00, domain=0x0000, type=pci, function=0x1}', managed='false', plugged='true', readOnly='false', deviceAlias='ide', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_16dad27f-1884-4f32-8736-6bbaf8cd2fbd=VmDevice:{id='VmDeviceId:{deviceId='16dad27f-1884-4f32-8736-6bbaf8cd2fbd', vmId='c64d80f9-a3bd-44f3-89c4-f1b9b509cdde'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=1}', managed='false', plugged='true', readOnly='false', deviceAlias='channel0', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}}, vmType=kvm, memSize=20480, smpCoresPerSocket=1, vmName=red, nice=0, status=WaitForLaunch, maxMemSize=81920, bootMenuEnable=false, pid=0, smpThreadsPerCore=1, memGuaranteedSize=16384, kvmEnable=true, displayNetwork=yorku, devices=[Ljava.lang.Object;@59fe293b, display=qxl, maxVCpus=16, clientIp=, statusTime=4295544360, maxMemSlots=16}, {acpiEnable=true, emulatedMachine=pc-i440fx-rhel7.3.0, vmId=1c7e966f-29e8-4788-b64c-86e52f415061, guestDiskMapping={}, transparentHugePages=true, timeOffset=0, cpuType=SandyBridge, smp=2, pauseCode=NOERR, guestNumaNodes=[Ljava.lang.Object;@570596a, smartcardEnable=false, custom={device_56041595-a1a1-4da5-9c82-0766a2a679e9device_caa0fc07-0456-4e86-ac52-e18524843fa8device_93d0045d-39ba-48e6-b704-3599b39f59f9device_5a114c4b-2119-4016-9a29-69f19deb09f1=VmDevice:{id='VmDeviceId:{deviceId='5a114c4b-2119-4016-9a29-69f19deb09f1', vmId='1c7e966f-29e8-4788-b64c-86e52f415061'}', device='ide', type='CONTROLLER', bootOrder='0', specParams='[]', address='{slot=0x01, bus=0x00, domain=0x0000, type=pci, function=0x1}', managed='false', plugged='true', readOnly='false', deviceAlias='ide', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_56041595-a1a1-4da5-9c82-0766a2a679e9device_caa0fc07-0456-4e86-ac52-e18524843fa8device_93d0045d-39ba-48e6-b704-3599b39f59f9=VmDevice:{id='VmDeviceId:{deviceId='93d0045d-39ba-48e6-b704-3599b39f59f9', vmId='1c7e966f-29e8-4788-b64c-86e52f415061'}', device='spicevmc', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=3}', managed='false', plugged='true', readOnly='false', deviceAlias='channel2', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_56041595-a1a1-4da5-9c82-0766a2a679e9=VmDevice:{id='VmDeviceId:{deviceId='56041595-a1a1-4da5-9c82-0766a2a679e9', vmId='1c7e966f-29e8-4788-b64c-86e52f415061'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=1}', managed='false', plugged='true', readOnly='false', deviceAlias='channel0', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_56041595-a1a1-4da5-9c82-0766a2a679e9device_caa0fc07-0456-4e86-ac52-e18524843fa8=VmDevice:{id='VmDeviceId:{deviceId='caa0fc07-0456-4e86-ac52-e18524843fa8', vmId='1c7e966f-29e8-4788-b64c-86e52f415061'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=2}', managed='false', plugged='true', readOnly='false', deviceAlias='channel1', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}}, vmType=kvm, memSize=4096, smpCoresPerSocket=1, vmName=udb, nice=0, status=Up, maxMemSize=16384, bootMenuEnable=false, pid=4865, smpThreadsPerCore=1, memGuaranteedSize=4096, kvmEnable=true, displayNetwork=yorku, devices=[Ljava.lang.Object;@4cb0fb01, display=qxl, maxVCpus=16, clientIp=, statusTime=4295544360, maxMemSlots=16}, {acpiEnable=true, emulatedMachine=pc-i440fx-rhel7.3.0, vmId=ae65a104-ed3f-40e0-ba92-5708cf338918, guestDiskMapping={}, transparentHugePages=true, timeOffset=0, cpuType=SandyBridge, smp=2, pauseCode=NOERR, guestNumaNodes=[Ljava.lang.Object;@ed61854, smartcardEnable=false, custom={device_ad0e60de-f05d-4a7d-beaa-50c74ff6e3b2device_caff46c6-9bc4-4657-82fa-74ddb1f395f3device_efaadb25-9ff8-4570-932c-97d8e90363cddevice_fd011cb3-5b43-4aa2-80ae-ac519e630280=VmDevice:{id='VmDeviceId:{deviceId='fd011cb3-5b43-4aa2-80ae-ac519e630280', vmId='ae65a104-ed3f-40e0-ba92-5708cf338918'}', device='ide', type='CONTROLLER', bootOrder='0', specParams='[]', address='{slot=0x01, bus=0x00, domain=0x0000, type=pci, function=0x1}', managed='false', plugged='true', readOnly='false', deviceAlias='ide', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_ad0e60de-f05d-4a7d-beaa-50c74ff6e3b2=VmDevice:{id='VmDeviceId:{deviceId='ad0e60de-f05d-4a7d-beaa-50c74ff6e3b2', vmId='ae65a104-ed3f-40e0-ba92-5708cf338918'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=1}', managed='false', plugged='true', readOnly='false', deviceAlias='channel0', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_ad0e60de-f05d-4a7d-beaa-50c74ff6e3b2device_caff46c6-9bc4-4657-82fa-74ddb1f395f3device_efaadb25-9ff8-4570-932c-97d8e90363cd=VmDevice:{id='VmDeviceId:{deviceId='efaadb25-9ff8-4570-932c-97d8e90363cd', vmId='ae65a104-ed3f-40e0-ba92-5708cf338918'}', device='spicevmc', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=3}', managed='false', plugged='true', readOnly='false', deviceAlias='channel2', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_ad0e60de-f05d-4a7d-beaa-50c74ff6e3b2device_caff46c6-9bc4-4657-82fa-74ddb1f395f3=VmDevice:{id='VmDeviceId:{deviceId='caff46c6-9bc4-4657-82fa-74ddb1f395f3', vmId='ae65a104-ed3f-40e0-ba92-5708cf338918'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=2}', managed='false', plugged='true', readOnly='false', deviceAlias='channel1', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}}, vmType=kvm, memSize=2048, smpCoresPerSocket=1, vmName=vpn2, nice=0, status=Up, maxMemSize=8192, bootMenuEnable=false, pid=5047, smpThreadsPerCore=1, memGuaranteedSize=1365, kvmEnable=true, displayNetwork=yorku, devices=[Ljava.lang.Object;@26c95f44, display=qxl, maxVCpus=16, clientIp=, statusTime=4295544370, maxMemSlots=16}], log id: 7b565bee 2017-10-01 16:35:11,202-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler8) [1f94017c] Received a memballoon Device without an address when processing VM c64d80f9-a3bd-44f3-89c4-f1b9b509cdde devices, skipping device: {device=memballoon, specParams={model=none}, type=balloon} 2017-10-01 16:35:11,203-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler8) [1f94017c] Received a virtio Device without an address when processing VM c64d80f9-a3bd-44f3-89c4-f1b9b509cdde devices, skipping device: {device=virtio, specParams={source=urandom}, model=virtio, type=rng, deviceId=e6cc13e5-c664-4435-a187-46d2b34ea993} 2017-10-01 16:35:11,203-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler8) [1f94017c] Received a spice Device without an address when processing VM c64d80f9-a3bd-44f3-89c4-f1b9b509cdde devices, skipping device: {device=spice, specParams={fileTransferEnable=true, displayNetwork=yorku, copyPasteEnable=true, displayIp=130.63.94.34, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,ssmartcard,susbredir}, type=graphics, deviceId=0000001f-001f-001f-001f-000000000268} 2017-10-01 16:35:11,204-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler8) [1f94017c] Received a memballoon Device without an address when processing VM 1c7e966f-29e8-4788-b64c-86e52f415061 devices, skipping device: {device=memballoon, specParams={model=none}, type=balloon, alias=balloon0} 2017-10-01 16:35:11,204-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler8) [1f94017c] Received a spice Device without an address when processing VM 1c7e966f-29e8-4788-b64c-86e52f415061 devices, skipping device: {device=spice, specParams={fileTransferEnable=true, displayNetwork=yorku, copyPasteEnable=true, displayIp=130.63.94.34, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,ssmartcard,susbredir}, type=graphics, deviceId=05aeb875-80f2-4a28-9070-4975d8dc2b9b, tlsPort=5901} 2017-10-01 16:35:11,205-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler8) [1f94017c] Received a spice Device without an address when processing VM ae65a104-ed3f-40e0-ba92-5708cf338918 devices, skipping device: {device=spice, specParams={fileTransferEnable=true, displayNetwork=yorku, copyPasteEnable=true, displayIp=130.63.94.34, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,ssmartcard,susbredir}, type=graphics, deviceId=f4fc08c8-a53e-48ef-be8e-6d847f92d837, tlsPort=5902} 2017-10-01 16:35:14,799-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmsStatisticsFetcher] (DefaultQuartzScheduler1) [b783b08] Fetched 9 VMs from VDS 'e1e71e9d-a0cd-47db-96c6-f724f6cea736' 2017-10-01 16:35:14,799-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler1) [b783b08] VM '07331c78-6e09-41b4-9b29-edda32f46351'(samba) moved from 'WaitForLaunch' --> 'PoweringUp' 2017-10-01 16:35:14,799-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler1) [b783b08] VM '1325cebf-834e-4571-a3e1-82beb0021ffe'(dns1) moved from 'WaitForLaunch' --> 'PoweringUp' 2017-10-01 16:35:14,800-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler1) [b783b08] VM 'cc1dbe93-bb80-451e-8b3b-c9cc5e90432f'(license) moved from 'WaitForLaunch' --> 'PoweringUp' 2017-10-01 16:35:14,800-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler1) [b783b08] VM '3bb83ad5-dc00-445d-8708-2db18c53e0e6'(www-vhost) moved from 'WaitForLaunch' --> 'PoweringUp' 2017-10-01 16:35:14,800-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler1) [b783b08] VM '59287796-c634-4f3c-94b1-2c9699986637'(vpn1) moved from 'WaitForLaunch' --> 'PoweringUp' 2017-10-01 16:35:14,800-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler1) [b783b08] VM 'd56363ad-9cb1-4a70-a33a-ce439b044d7d'(svn) moved from 'WaitForLaunch' --> 'PoweringUp' 2017-10-01 16:35:14,820-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVDSCommand] (DefaultQuartzScheduler1) [b783b08] START, FullListVDSCommand(HostName = virt3, FullListVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vmIds='[cc1dbe93-bb80-451e-8b3b-c9cc5e90432f, d56363ad-9cb1-4a70-a33a-ce439b044d7d, d66235b4-7c8d-4da2-a572-82ecc9c53a59, 07331c78-6e09-41b4-9b29-edda32f46351, 1325cebf-834e-4571-a3e1-82beb0021ffe, 3bb83ad5-dc00-445d-8708-2db18c53e0e6, 59287796-c634-4f3c-94b1-2c9699986637]'}), log id: 42f5bd64 2017-10-01 16:35:14,935-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVDSCommand] (DefaultQuartzScheduler1) [b783b08] FINISH, FullListVDSCommand, return: [{acpiEnable=true, emulatedMachine=pc-i440fx-rhel7.3.0, vmId=d56363ad-9cb1-4a70-a33a-ce439b044d7d, guestDiskMapping={}, transparentHugePages=true, timeOffset=0, cpuType=SandyBridge, smp=4, pauseCode=NOERR, guestNumaNodes=[Ljava.lang.Object;@7b12c200, smartcardEnable=false, custom={device_b4a86393-6dfb-4d30-9cb2-2174ee1b1ad3=VmDevice:{id='VmDeviceId:{deviceId='b4a86393-6dfb-4d30-9cb2-2174ee1b1ad3', vmId='d56363ad-9cb1-4a70-a33a-ce439b044d7d'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=1}', managed='false', plugged='true', readOnly='false', deviceAlias='channel0', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_b4a86393-6dfb-4d30-9cb2-2174ee1b1ad3device_58b4d0dc-c3f5-4ce5-888d-d6215a4a982e=VmDevice:{id='VmDeviceId:{deviceId='58b4d0dc-c3f5-4ce5-888d-d6215a4a982e', vmId='d56363ad-9cb1-4a70-a33a-ce439b044d7d'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=2}', managed='false', plugged='true', readOnly='false', deviceAlias='channel1', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_b4a86393-6dfb-4d30-9cb2-2174ee1b1ad3device_58b4d0dc-c3f5-4ce5-888d-d6215a4a982edevice_0e9f02f4-7590-4b92-a564-6314086bc158=VmDevice:{id='VmDeviceId:{deviceId='0e9f02f4-7590-4b92-a564-6314086bc158', vmId='d56363ad-9cb1-4a70-a33a-ce439b044d7d'}', device='spicevmc', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=3}', managed='false', plugged='true', readOnly='false', deviceAlias='channel2', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_b4a86393-6dfb-4d30-9cb2-2174ee1b1ad3device_58b4d0dc-c3f5-4ce5-888d-d6215a4a982edevice_0e9f02f4-7590-4b92-a564-6314086bc158device_956e0903-a772-41d8-bf74-42325bc04f6f=VmDevice:{id='VmDeviceId:{deviceId='956e0903-a772-41d8-bf74-42325bc04f6f', vmId='d56363ad-9cb1-4a70-a33a-ce439b044d7d'}', device='ide', type='CONTROLLER', bootOrder='0', specParams='[]', address='{slot=0x01, bus=0x00, domain=0x0000, type=pci, function=0x1}', managed='false', plugged='true', readOnly='false', deviceAlias='ide', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}}, vmType=kvm, memSize=4096, smpCoresPerSocket=1, vmName=svn, nice=0, status=Up, maxMemSize=16384, bootMenuEnable=false, pid=7060, smpThreadsPerCore=1, memGuaranteedSize=4096, kvmEnable=true, displayNetwork=yorku, devices=[Ljava.lang.Object;@7518d8f3, display=qxl, maxVCpus=16, clientIp=, statusTime=5262731280, maxMemSlots=16}, {acpiEnable=true, emulatedMachine=pc-i440fx-rhel7.3.0, vmId=d66235b4-7c8d-4da2-a572-82ecc9c53a59, guestDiskMapping={}, transparentHugePages=true, timeOffset=0, cpuType=SandyBridge, smp=4, guestNumaNodes=[Ljava.lang.Object;@f5bb816, smartcardEnable=false, custom={device_e6f104af-4245-4a6d-8c1d-ecc3120daaabdevice_783850c4-36cd-49c5-b1b7-1b94ad859083=VmDevice:{id='VmDeviceId:{deviceId='783850c4-36cd-49c5-b1b7-1b94ad859083', vmId='d66235b4-7c8d-4da2-a572-82ecc9c53a59'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=2}', managed='false', plugged='true', readOnly='false', deviceAlias='channel1', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_e6f104af-4245-4a6d-8c1d-ecc3120daaabdevice_783850c4-36cd-49c5-b1b7-1b94ad859083device_d94441b2-fa26-4878-b6d1-468c3d8cc29edevice_905471fd-92ff-4c8a-ae24-c8b018279497=VmDevice:{id='VmDeviceId:{deviceId='905471fd-92ff-4c8a-ae24-c8b018279497', vmId='d66235b4-7c8d-4da2-a572-82ecc9c53a59'}', device='ide', type='CONTROLLER', bootOrder='0', specParams='[]', address='{slot=0x01, bus=0x00, domain=0x0000, type=pci, function=0x1}', managed='false', plugged='true', readOnly='false', deviceAlias='ide', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_e6f104af-4245-4a6d-8c1d-ecc3120daaab=VmDevice:{id='VmDeviceId:{deviceId='e6f104af-4245-4a6d-8c1d-ecc3120daaab', vmId='d66235b4-7c8d-4da2-a572-82ecc9c53a59'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=1}', managed='false', plugged='true', readOnly='false', deviceAlias='channel0', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_e6f104af-4245-4a6d-8c1d-ecc3120daaabdevice_783850c4-36cd-49c5-b1b7-1b94ad859083device_d94441b2-fa26-4878-b6d1-468c3d8cc29e=VmDevice:{id='VmDeviceId:{deviceId='d94441b2-fa26-4878-b6d1-468c3d8cc29e', vmId='d66235b4-7c8d-4da2-a572-82ecc9c53a59'}', device='spicevmc', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=3}', managed='false', plugged='true', readOnly='false', deviceAlias='channel2', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}}, vmType=kvm, memSize=4096, smpCoresPerSocket=1, vmName=www-las, nice=0, status=WaitForLaunch, maxMemSize=16384, bootMenuEnable=false, pid=0, smpThreadsPerCore=1, memGuaranteedSize=4096, kvmEnable=true, displayNetwork=yorku, devices=[Ljava.lang.Object;@60aa2a04, display=qxl, maxVCpus=16, clientIp=, statusTime=5262731280, maxMemSlots=16}, {acpiEnable=true, emulatedMachine=pc-i440fx-rhel7.3.0, vmId=1325cebf-834e-4571-a3e1-82beb0021ffe, guestDiskMapping={}, transparentHugePages=true, timeOffset=0, cpuType=SandyBridge, smp=1, pauseCode=NOERR, guestNumaNodes=[Ljava.lang.Object;@dc5e6d3, smartcardEnable=false, custom={device_e8518a41-75b4-4047-9698-18caf5fc06f0device_fc446988-d249-4f93-80f6-c15a4ae3b8d9=VmDevice:{id='VmDeviceId:{deviceId='fc446988-d249-4f93-80f6-c15a4ae3b8d9', vmId='1325cebf-834e-4571-a3e1-82beb0021ffe'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=2}', managed='false', plugged='true', readOnly='false', deviceAlias='channel1', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_e8518a41-75b4-4047-9698-18caf5fc06f0device_fc446988-d249-4f93-80f6-c15a4ae3b8d9device_fca5daed-4de2-4882-b244-d15786b5267ddevice_9690f494-21ec-440d-ad07-62002944e0a7=VmDevice:{id='VmDeviceId:{deviceId='9690f494-21ec-440d-ad07-62002944e0a7', vmId='1325cebf-834e-4571-a3e1-82beb0021ffe'}', device='ide', type='CONTROLLER', bootOrder='0', specParams='[]', address='{slot=0x01, bus=0x00, domain=0x0000, type=pci, function=0x1}', managed='false', plugged='true', readOnly='false', deviceAlias='ide', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_e8518a41-75b4-4047-9698-18caf5fc06f0device_fc446988-d249-4f93-80f6-c15a4ae3b8d9device_fca5daed-4de2-4882-b244-d15786b5267d=VmDevice:{id='VmDeviceId:{deviceId='fca5daed-4de2-4882-b244-d15786b5267d', vmId='1325cebf-834e-4571-a3e1-82beb0021ffe'}', device='spicevmc', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=3}', managed='false', plugged='true', readOnly='false', deviceAlias='channel2', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_e8518a41-75b4-4047-9698-18caf5fc06f0=VmDevice:{id='VmDeviceId:{deviceId='e8518a41-75b4-4047-9698-18caf5fc06f0', vmId='1325cebf-834e-4571-a3e1-82beb0021ffe'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=1}', managed='false', plugged='true', readOnly='false', deviceAlias='channel0', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}}, vmType=kvm, memSize=2048, smpCoresPerSocket=1, vmName=dns1, nice=0, status=Up, maxMemSize=8192, bootMenuEnable=false, pid=7407, smpThreadsPerCore=1, memGuaranteedSize=1365, kvmEnable=true, displayNetwork=yorku, devices=[Ljava.lang.Object;@3ea978d8, display=qxl, maxVCpus=16, clientIp=, statusTime=5262731280, maxMemSlots=16}, {acpiEnable=true, emulatedMachine=pc-i440fx-rhel7.3.0, vmId=07331c78-6e09-41b4-9b29-edda32f46351, guestDiskMapping={}, transparentHugePages=true, timeOffset=0, cpuType=SandyBridge, smp=4, pauseCode=NOERR, guestNumaNodes=[Ljava.lang.Object;@6c60fc13, smartcardEnable=false, custom={device_f9fabb45-65d0-44aa-8519-442f0490daabdevice_720907bd-38e0-4e2e-82b3-0c09e2d8c76e=VmDevice:{id='VmDeviceId:{deviceId='720907bd-38e0-4e2e-82b3-0c09e2d8c76e', vmId='07331c78-6e09-41b4-9b29-edda32f46351'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=2}', managed='false', plugged='true', readOnly='false', deviceAlias='channel1', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_f9fabb45-65d0-44aa-8519-442f0490daab=VmDevice:{id='VmDeviceId:{deviceId='f9fabb45-65d0-44aa-8519-442f0490daab', vmId='07331c78-6e09-41b4-9b29-edda32f46351'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=1}', managed='false', plugged='true', readOnly='false', deviceAlias='channel0', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_f9fabb45-65d0-44aa-8519-442f0490daabdevice_720907bd-38e0-4e2e-82b3-0c09e2d8c76edevice_4c7ee575-6e86-4fe7-91ed-3c9cf3a71bb5=VmDevice:{id='VmDeviceId:{deviceId='4c7ee575-6e86-4fe7-91ed-3c9cf3a71bb5', vmId='07331c78-6e09-41b4-9b29-edda32f46351'}', device='spicevmc', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=3}', managed='false', plugged='true', readOnly='false', deviceAlias='channel2', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_f9fabb45-65d0-44aa-8519-442f0490daabdevice_720907bd-38e0-4e2e-82b3-0c09e2d8c76edevice_4c7ee575-6e86-4fe7-91ed-3c9cf3a71bb5device_1ee85fb3-575f-4aa6-8664-16938022b9e2=VmDevice:{id='VmDeviceId:{deviceId='1ee85fb3-575f-4aa6-8664-16938022b9e2', vmId='07331c78-6e09-41b4-9b29-edda32f46351'}', device='ide', type='CONTROLLER', bootOrder='0', specParams='[]', address='{slot=0x01, bus=0x00, domain=0x0000, type=pci, function=0x1}', managed='false', plugged='true', readOnly='false', deviceAlias='ide', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}}, vmType=kvm, memSize=4096, smpCoresPerSocket=1, vmName=samba, nice=0, status=Up, maxMemSize=16384, bootMenuEnable=false, pid=7235, smpThreadsPerCore=1, memGuaranteedSize=4096, kvmEnable=true, displayNetwork=yorku, devices=[Ljava.lang.Object;@5dab1191, display=qxl, maxVCpus=16, clientIp=, statusTime=5262731280, maxMemSlots=16}, {acpiEnable=true, emulatedMachine=pc-i440fx-rhel7.3.0, vmId=cc1dbe93-bb80-451e-8b3b-c9cc5e90432f, guestDiskMapping={}, transparentHugePages=true, timeOffset=0, cpuType=SandyBridge, smp=2, pauseCode=NOERR, guestNumaNodes=[Ljava.lang.Object;@7f7a8789, smartcardEnable=false, custom={device_c6badb18-1587-4042-8dbb-08b45a345f53=VmDevice:{id='VmDeviceId:{deviceId='c6badb18-1587-4042-8dbb-08b45a345f53', vmId='cc1dbe93-bb80-451e-8b3b-c9cc5e90432f'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=1}', managed='false', plugged='true', readOnly='false', deviceAlias='channel0', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_c6badb18-1587-4042-8dbb-08b45a345f53device_cf1ac984-b593-4de6-8b5a-332e8f931f50=VmDevice:{id='VmDeviceId:{deviceId='cf1ac984-b593-4de6-8b5a-332e8f931f50', vmId='cc1dbe93-bb80-451e-8b3b-c9cc5e90432f'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=2}', managed='false', plugged='true', readOnly='false', deviceAlias='channel1', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_c6badb18-1587-4042-8dbb-08b45a345f53device_cf1ac984-b593-4de6-8b5a-332e8f931f50device_024fe828-a608-42f1-8e96-6425c35b0690=VmDevice:{id='VmDeviceId:{deviceId='024fe828-a608-42f1-8e96-6425c35b0690', vmId='cc1dbe93-bb80-451e-8b3b-c9cc5e90432f'}', device='spicevmc', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=3}', managed='false', plugged='true', readOnly='false', deviceAlias='channel2', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_c6badb18-1587-4042-8dbb-08b45a345f53device_cf1ac984-b593-4de6-8b5a-332e8f931f50device_024fe828-a608-42f1-8e96-6425c35b0690device_b1e2b19c-73b7-494d-9186-12d039cc81ca=VmDevice:{id='VmDeviceId:{deviceId='b1e2b19c-73b7-494d-9186-12d039cc81ca', vmId='cc1dbe93-bb80-451e-8b3b-c9cc5e90432f'}', device='ide', type='CONTROLLER', bootOrder='0', specParams='[]', address='{slot=0x01, bus=0x00, domain=0x0000, type=pci, function=0x1}', managed='false', plugged='true', readOnly='false', deviceAlias='ide', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}}, vmType=kvm, memSize=4096, smpCoresPerSocket=1, vmName=license, nice=0, status=Up, maxMemSize=16384, bootMenuEnable=false, pid=7943, smpThreadsPerCore=1, memGuaranteedSize=2730, kvmEnable=true, displayNetwork=yorku, devices=[Ljava.lang.Object;@ae04f06, display=qxl, maxVCpus=16, clientIp=, statusTime=5262731280, maxMemSlots=16}, {acpiEnable=true, emulatedMachine=pc-i440fx-rhel7.3.0, vmId=3bb83ad5-dc00-445d-8708-2db18c53e0e6, guestDiskMapping={}, transparentHugePages=true, timeOffset=0, cpuType=SandyBridge, smp=4, pauseCode=NOERR, guestNumaNodes=[Ljava.lang.Object;@618baa2d, smartcardEnable=false, custom={device_53c485bb-d380-4a20-b5cb-cc26b577713fdevice_11c0a69b-6d65-4b6f-8a60-955245903452=VmDevice:{id='VmDeviceId:{deviceId='11c0a69b-6d65-4b6f-8a60-955245903452', vmId='3bb83ad5-dc00-445d-8708-2db18c53e0e6'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=2}', managed='false', plugged='true', readOnly='false', deviceAlias='channel1', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_53c485bb-d380-4a20-b5cb-cc26b577713f=VmDevice:{id='VmDeviceId:{deviceId='53c485bb-d380-4a20-b5cb-cc26b577713f', vmId='3bb83ad5-dc00-445d-8708-2db18c53e0e6'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=1}', managed='false', plugged='true', readOnly='false', deviceAlias='channel0', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_53c485bb-d380-4a20-b5cb-cc26b577713fdevice_11c0a69b-6d65-4b6f-8a60-955245903452device_e2ed8694-2c47-49a9-ada0-1f7e225e55f1device_8eb8899b-6d1c-4062-b558-3c39463747d9=VmDevice:{id='VmDeviceId:{deviceId='8eb8899b-6d1c-4062-b558-3c39463747d9', vmId='3bb83ad5-dc00-445d-8708-2db18c53e0e6'}', device='ide', type='CONTROLLER', bootOrder='0', specParams='[]', address='{slot=0x01, bus=0x00, domain=0x0000, type=pci, function=0x1}', managed='false', plugged='true', readOnly='false', deviceAlias='ide', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_53c485bb-d380-4a20-b5cb-cc26b577713fdevice_11c0a69b-6d65-4b6f-8a60-955245903452device_e2ed8694-2c47-49a9-ada0-1f7e225e55f1=VmDevice:{id='VmDeviceId:{deviceId='e2ed8694-2c47-49a9-ada0-1f7e225e55f1', vmId='3bb83ad5-dc00-445d-8708-2db18c53e0e6'}', device='spicevmc', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=3}', managed='false', plugged='true', readOnly='false', deviceAlias='channel2', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}}, vmType=kvm, memSize=4096, smpCoresPerSocket=1, vmName=www-vhost, nice=0, status=Up, maxMemSize=16384, bootMenuEnable=false, pid=7804, smpThreadsPerCore=1, memGuaranteedSize=4096, kvmEnable=true, displayNetwork=yorku, devices=[Ljava.lang.Object;@64290814, display=qxl, maxVCpus=16, clientIp=, statusTime=5262731280, maxMemSlots=16}, {acpiEnable=true, emulatedMachine=pc-i440fx-rhel7.3.0, vmId=59287796-c634-4f3c-94b1-2c9699986637, guestDiskMapping={}, transparentHugePages=true, timeOffset=0, cpuType=SandyBridge, smp=2, pauseCode=NOERR, guestNumaNodes=[Ljava.lang.Object;@4f448dd7, smartcardEnable=false, custom={device_69e94312-87d3-493e-9665-92afdfe15039device_10796316-52cc-4731-9300-9f72ef554368device_5034296d-1ccc-4781-9aa0-f0677e556ed6device_1ef0ef51-10e7-436c-9fb7-dd73bd03cf40=VmDevice:{id='VmDeviceId:{deviceId='1ef0ef51-10e7-436c-9fb7-dd73bd03cf40', vmId='59287796-c634-4f3c-94b1-2c9699986637'}', device='ide', type='CONTROLLER', bootOrder='0', specParams='[]', address='{slot=0x01, bus=0x00, domain=0x0000, type=pci, function=0x1}', managed='false', plugged='true', readOnly='false', deviceAlias='ide', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_69e94312-87d3-493e-9665-92afdfe15039=VmDevice:{id='VmDeviceId:{deviceId='69e94312-87d3-493e-9665-92afdfe15039', vmId='59287796-c634-4f3c-94b1-2c9699986637'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=1}', managed='false', plugged='true', readOnly='false', deviceAlias='channel0', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_69e94312-87d3-493e-9665-92afdfe15039device_10796316-52cc-4731-9300-9f72ef554368device_5034296d-1ccc-4781-9aa0-f0677e556ed6=VmDevice:{id='VmDeviceId:{deviceId='5034296d-1ccc-4781-9aa0-f0677e556ed6', vmId='59287796-c634-4f3c-94b1-2c9699986637'}', device='spicevmc', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=3}', managed='false', plugged='true', readOnly='false', deviceAlias='channel2', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_69e94312-87d3-493e-9665-92afdfe15039device_10796316-52cc-4731-9300-9f72ef554368=VmDevice:{id='VmDeviceId:{deviceId='10796316-52cc-4731-9300-9f72ef554368', vmId='59287796-c634-4f3c-94b1-2c9699986637'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=2}', managed='false', plugged='true', readOnly='false', deviceAlias='channel1', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}}, vmType=kvm, memSize=2048, smpCoresPerSocket=1, vmName=vpn1, nice=0, status=Up, maxMemSize=8192, bootMenuEnable=false, pid=7626, smpThreadsPerCore=1, memGuaranteedSize=1365, kvmEnable=true, displayNetwork=yorku, devices=[Ljava.lang.Object;@47d46346, display=qxl, maxVCpus=16, clientIp=, statusTime=5262731280, maxMemSlots=16}], log id: 42f5bd64 2017-10-01 16:35:14,936-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler1) [b783b08] Received a memballoon Device without an address when processing VM d56363ad-9cb1-4a70-a33a-ce439b044d7d devices, skipping device: {device=memballoon, specParams={model=none}, type=balloon, alias=balloon0} 2017-10-01 16:35:14,936-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler1) [b783b08] Received a spice Device without an address when processing VM d56363ad-9cb1-4a70-a33a-ce439b044d7d devices, skipping device: {device=spice, specParams={fileTransferEnable=true, displayNetwork=yorku, copyPasteEnable=true, displayIp=130.63.94.36, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,ssmartcard,susbredir}, type=graphics, deviceId=0000002e-002e-002e-002e-00000000036d, tlsPort=5902} 2017-10-01 16:35:14,936-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler1) [b783b08] Received a memballoon Device without an address when processing VM d66235b4-7c8d-4da2-a572-82ecc9c53a59 devices, skipping device: {device=memballoon, specParams={model=none}, type=balloon} 2017-10-01 16:35:14,936-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler1) [b783b08] Received a virtio Device without an address when processing VM d66235b4-7c8d-4da2-a572-82ecc9c53a59 devices, skipping device: {device=virtio, specParams={source=urandom}, model=virtio, type=rng, deviceId=4440cf64-af9a-4390-a6eb-504a8c03a12a} 2017-10-01 16:35:14,937-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler1) [b783b08] Received a spice Device without an address when processing VM d66235b4-7c8d-4da2-a572-82ecc9c53a59 devices, skipping device: {device=spice, specParams={fileTransferEnable=true, displayNetwork=yorku, copyPasteEnable=true, displayIp=130.63.94.36, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,ssmartcard,susbredir}, type=graphics, deviceId=00000032-0032-0032-0032-0000000002c8} 2017-10-01 16:35:14,937-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler1) [b783b08] Received a memballoon Device without an address when processing VM 1325cebf-834e-4571-a3e1-82beb0021ffe devices, skipping device: {device=memballoon, specParams={model=none}, type=balloon, alias=balloon0} 2017-10-01 16:35:14,937-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler1) [b783b08] Received a spice Device without an address when processing VM 1325cebf-834e-4571-a3e1-82beb0021ffe devices, skipping device: {device=spice, specParams={fileTransferEnable=true, displayNetwork=yorku, copyPasteEnable=true, displayIp=130.63.94.36, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,ssmartcard,susbredir}, type=graphics, deviceId=3f758753-b44e-4641-a51c-e0554c983f33, tlsPort=5904} 2017-10-01 16:35:14,938-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler1) [b783b08] Received a memballoon Device without an address when processing VM 07331c78-6e09-41b4-9b29-edda32f46351 devices, skipping device: {device=memballoon, specParams={model=none}, type=balloon, alias=balloon0} 2017-10-01 16:35:14,938-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler1) [b783b08] Received a spice Device without an address when processing VM 07331c78-6e09-41b4-9b29-edda32f46351 devices, skipping device: {device=spice, specParams={fileTransferEnable=true, displayNetwork=yorku, copyPasteEnable=true, displayIp=130.63.94.36, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,ssmartcard,susbredir}, type=graphics, deviceId=00000033-0033-0033-0033-00000000028f, tlsPort=5903} 2017-10-01 16:35:14,939-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler1) [b783b08] Received a memballoon Device without an address when processing VM cc1dbe93-bb80-451e-8b3b-c9cc5e90432f devices, skipping device: {device=memballoon, specParams={model=none}, type=balloon, alias=balloon0} 2017-10-01 16:35:14,939-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler1) [b783b08] Received a spice Device without an address when processing VM cc1dbe93-bb80-451e-8b3b-c9cc5e90432f devices, skipping device: {device=spice, specParams={fileTransferEnable=true, displayNetwork=yorku, copyPasteEnable=true, displayIp=130.63.94.36, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,ssmartcard,susbredir}, type=graphics, deviceId=6fd350b3-e6f6-46ec-981d-d0250aebb024, tlsPort=5907} 2017-10-01 16:35:14,939-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler1) [b783b08] Received a memballoon Device without an address when processing VM 3bb83ad5-dc00-445d-8708-2db18c53e0e6 devices, skipping device: {device=memballoon, specParams={model=none}, type=balloon, alias=balloon0} 2017-10-01 16:35:14,939-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler1) [b783b08] Received a spice Device without an address when processing VM 3bb83ad5-dc00-445d-8708-2db18c53e0e6 devices, skipping device: {device=spice, specParams={fileTransferEnable=true, displayNetwork=yorku, copyPasteEnable=true, displayIp=130.63.94.36, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,ssmartcard,susbredir}, type=graphics, deviceId=0000002b-002b-002b-002b-000000000192, tlsPort=5906} 2017-10-01 16:35:14,940-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler1) [b783b08] Received a spice Device without an address when processing VM 59287796-c634-4f3c-94b1-2c9699986637 devices, skipping device: {device=spice, specParams={fileTransferEnable=true, displayNetwork=yorku, copyPasteEnable=true, displayIp=130.63.94.36, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,ssmartcard,susbredir}, type=graphics, deviceId=22da7b7b-8a47-4039-b926-d18882ea2f9d, tlsPort=5905} 2017-10-01 16:35:21,265-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler4) [2006068] VM '1734fed5-fc97-4714-bb63-55e8f4eb6d34'(wiki) moved from 'WaitForLaunch' --> 'PoweringUp' 2017-10-01 16:35:21,265-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler4) [2006068] VM '9ea2821e-557f-45cb-9872-1e6c5aeaf246'(rs) moved from 'WaitForLaunch' --> 'PoweringUp' 2017-10-01 16:35:21,275-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVDSCommand] (DefaultQuartzScheduler4) [2006068] START, FullListVDSCommand(HostName = virt2, FullListVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vmIds='[9ea2821e-557f-45cb-9872-1e6c5aeaf246, 1734fed5-fc97-4714-bb63-55e8f4eb6d34]'}), log id: 4f1e9133 2017-10-01 16:35:21,301-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVDSCommand] (DefaultQuartzScheduler4) [2006068] FINISH, FullListVDSCommand, return: [{acpiEnable=true, emulatedMachine=pc-i440fx-rhel7.3.0, vmId=1734fed5-fc97-4714-bb63-55e8f4eb6d34, guestDiskMapping={}, transparentHugePages=true, timeOffset=0, cpuType=SandyBridge, smp=2, pauseCode=NOERR, guestNumaNodes=[Ljava.lang.Object;@25c0a40d, smartcardEnable=false, custom={device_9d496570-3467-475b-ba9c-3111478eb9a7device_fb1efd5f-e403-44f3-ae46-9e817b19ee91=VmDevice:{id='VmDeviceId:{deviceId='fb1efd5f-e403-44f3-ae46-9e817b19ee91', vmId='1734fed5-fc97-4714-bb63-55e8f4eb6d34'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=2}', managed='false', plugged='true', readOnly='false', deviceAlias='channel1', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_9d496570-3467-475b-ba9c-3111478eb9a7device_fb1efd5f-e403-44f3-ae46-9e817b19ee91device_953c69bc-7182-496a-8180-1a8dc3230464device_9a7950a3-fb9b-4f82-adac-d8246f775af0=VmDevice:{id='VmDeviceId:{deviceId='9a7950a3-fb9b-4f82-adac-d8246f775af0', vmId='1734fed5-fc97-4714-bb63-55e8f4eb6d34'}', device='ide', type='CONTROLLER', bootOrder='0', specParams='[]', address='{slot=0x01, bus=0x00, domain=0x0000, type=pci, function=0x1}', managed='false', plugged='true', readOnly='false', deviceAlias='ide', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_9d496570-3467-475b-ba9c-3111478eb9a7=VmDevice:{id='VmDeviceId:{deviceId='9d496570-3467-475b-ba9c-3111478eb9a7', vmId='1734fed5-fc97-4714-bb63-55e8f4eb6d34'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=1}', managed='false', plugged='true', readOnly='false', deviceAlias='channel0', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_9d496570-3467-475b-ba9c-3111478eb9a7device_fb1efd5f-e403-44f3-ae46-9e817b19ee91device_953c69bc-7182-496a-8180-1a8dc3230464=VmDevice:{id='VmDeviceId:{deviceId='953c69bc-7182-496a-8180-1a8dc3230464', vmId='1734fed5-fc97-4714-bb63-55e8f4eb6d34'}', device='spicevmc', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=3}', managed='false', plugged='true', readOnly='false', deviceAlias='channel2', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}}, vmType=kvm, memSize=4096, smpCoresPerSocket=1, vmName=wiki, nice=0, status=Up, maxMemSize=16384, bootMenuEnable=false, pid=3708, smpThreadsPerCore=1, memGuaranteedSize=4096, kvmEnable=true, displayNetwork=yorku, devices=[Ljava.lang.Object;@335d1fde, display=qxl, maxVCpus=16, clientIp=, statusTime=4726804510, maxMemSlots=16}, {acpiEnable=true, emulatedMachine=pc-i440fx-rhel7.3.0, vmId=9ea2821e-557f-45cb-9872-1e6c5aeaf246, guestDiskMapping={}, transparentHugePages=true, timeOffset=0, cpuType=SandyBridge, smp=1, pauseCode=NOERR, guestNumaNodes=[Ljava.lang.Object;@7f7d121f, smartcardEnable=false, custom={device_ede985f5-ebdf-4058-9bd6-85b32f6086d1device_210d38b2-63a2-4b1e-8d59-3ede51837fc8=VmDevice:{id='VmDeviceId:{deviceId='210d38b2-63a2-4b1e-8d59-3ede51837fc8', vmId='9ea2821e-557f-45cb-9872-1e6c5aeaf246'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=2}', managed='false', plugged='true', readOnly='false', deviceAlias='channel1', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_ede985f5-ebdf-4058-9bd6-85b32f6086d1device_210d38b2-63a2-4b1e-8d59-3ede51837fc8device_b674e26e-33c4-45f6-b6d6-a751958e7889device_64356cea-121e-49a6-b2e3-b61cf6ef581d=VmDevice:{id='VmDeviceId:{deviceId='64356cea-121e-49a6-b2e3-b61cf6ef581d', vmId='9ea2821e-557f-45cb-9872-1e6c5aeaf246'}', device='ide', type='CONTROLLER', bootOrder='0', specParams='[]', address='{slot=0x01, bus=0x00, domain=0x0000, type=pci, function=0x1}', managed='false', plugged='true', readOnly='false', deviceAlias='ide', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_ede985f5-ebdf-4058-9bd6-85b32f6086d1device_210d38b2-63a2-4b1e-8d59-3ede51837fc8device_b674e26e-33c4-45f6-b6d6-a751958e7889=VmDevice:{id='VmDeviceId:{deviceId='b674e26e-33c4-45f6-b6d6-a751958e7889', vmId='9ea2821e-557f-45cb-9872-1e6c5aeaf246'}', device='spicevmc', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=3}', managed='false', plugged='true', readOnly='false', deviceAlias='channel2', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_ede985f5-ebdf-4058-9bd6-85b32f6086d1=VmDevice:{id='VmDeviceId:{deviceId='ede985f5-ebdf-4058-9bd6-85b32f6086d1', vmId='9ea2821e-557f-45cb-9872-1e6c5aeaf246'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=1}', managed='false', plugged='true', readOnly='false', deviceAlias='channel0', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}}, vmType=kvm, memSize=1024, smpCoresPerSocket=1, vmName=rs, nice=0, status=Up, maxMemSize=4096, bootMenuEnable=false, pid=3527, smpThreadsPerCore=1, memGuaranteedSize=682, kvmEnable=true, displayNetwork=yorku, devices=[Ljava.lang.Object;@44d183ca, display=qxl, maxVCpus=16, clientIp=, statusTime=4726804510, maxMemSlots=16}], log id: 4f1e9133 2017-10-01 16:35:21,302-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler4) [2006068] Received a memballoon Device without an address when processing VM 1734fed5-fc97-4714-bb63-55e8f4eb6d34 devices, skipping device: {device=memballoon, specParams={model=none}, type=balloon, alias=balloon0} 2017-10-01 16:35:21,302-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler4) [2006068] Received a spice Device without an address when processing VM 1734fed5-fc97-4714-bb63-55e8f4eb6d34 devices, skipping device: {device=spice, specParams={fileTransferEnable=true, displayNetwork=yorku, copyPasteEnable=true, displayIp=130.63.94.35, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,ssmartcard,susbredir}, type=graphics, deviceId=0000002d-002d-002d-002d-0000000001ac, tlsPort=5904} 2017-10-01 16:35:21,302-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler4) [2006068] Received a memballoon Device without an address when processing VM 9ea2821e-557f-45cb-9872-1e6c5aeaf246 devices, skipping device: {device=memballoon, specParams={model=none}, type=balloon, alias=balloon0} 2017-10-01 16:35:21,302-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler4) [2006068] Received a spice Device without an address when processing VM 9ea2821e-557f-45cb-9872-1e6c5aeaf246 devices, skipping device: {device=spice, specParams={fileTransferEnable=true, displayNetwork=yorku, copyPasteEnable=true, displayIp=130.63.94.35, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,ssmartcard,susbredir}, type=graphics, deviceId=58c8cadc-6a62-488c-85d2-dc06c41907f3, tlsPort=5903} 2017-10-01 16:35:25,441-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler6) [485f9af] VM 'ae56c31c-71f7-433a-b198-a1c2c846325e'(forum) moved from 'WaitForLaunch' --> 'PoweringUp' 2017-10-01 16:35:25,467-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVDSCommand] (DefaultQuartzScheduler6) [485f9af] START, FullListVDSCommand(HostName = virt4, FullListVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vmIds='[ae56c31c-71f7-433a-b198-a1c2c846325e]'}), log id: 2808a05a 2017-10-01 16:35:25,478-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVDSCommand] (DefaultQuartzScheduler6) [485f9af] FINISH, FullListVDSCommand, return: [{acpiEnable=true, emulatedMachine=pc-i440fx-rhel7.3.0, vmId=ae56c31c-71f7-433a-b198-a1c2c846325e, guestDiskMapping={}, transparentHugePages=true, timeOffset=0, cpuType=SandyBridge, smp=2, pauseCode=NOERR, guestNumaNodes=[Ljava.lang.Object;@17febcf1, smartcardEnable=false, custom={device_80a61fdb-b84a-4c66-bbcc-4a3a257b583f=VmDevice:{id='VmDeviceId:{deviceId='80a61fdb-b84a-4c66-bbcc-4a3a257b583f', vmId='ae56c31c-71f7-433a-b198-a1c2c846325e'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=1}', managed='false', plugged='true', readOnly='false', deviceAlias='channel0', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_80a61fdb-b84a-4c66-bbcc-4a3a257b583fdevice_5f697df1-cc6b-402f-8b25-ad143b5ee816device_f660a99d-ec1f-4a44-8f99-ae21606bfd34device_62c8af1e-b59b-4bd8-99dd-40948e53b6df=VmDevice:{id='VmDeviceId:{deviceId='62c8af1e-b59b-4bd8-99dd-40948e53b6df', vmId='ae56c31c-71f7-433a-b198-a1c2c846325e'}', device='ide', type='CONTROLLER', bootOrder='0', specParams='[]', address='{slot=0x01, bus=0x00, domain=0x0000, type=pci, function=0x1}', managed='false', plugged='true', readOnly='false', deviceAlias='ide', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_80a61fdb-b84a-4c66-bbcc-4a3a257b583fdevice_5f697df1-cc6b-402f-8b25-ad143b5ee816device_f660a99d-ec1f-4a44-8f99-ae21606bfd34=VmDevice:{id='VmDeviceId:{deviceId='f660a99d-ec1f-4a44-8f99-ae21606bfd34', vmId='ae56c31c-71f7-433a-b198-a1c2c846325e'}', device='spicevmc', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=3}', managed='false', plugged='true', readOnly='false', deviceAlias='channel2', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_80a61fdb-b84a-4c66-bbcc-4a3a257b583fdevice_5f697df1-cc6b-402f-8b25-ad143b5ee816=VmDevice:{id='VmDeviceId:{deviceId='5f697df1-cc6b-402f-8b25-ad143b5ee816', vmId='ae56c31c-71f7-433a-b198-a1c2c846325e'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=2}', managed='false', plugged='true', readOnly='false', deviceAlias='channel1', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}}, vmType=kvm, memSize=4096, smpCoresPerSocket=1, vmName=forum, nice=0, status=Up, maxMemSize=16384, bootMenuEnable=false, pid=108388, smpThreadsPerCore=1, memGuaranteedSize=4096, kvmEnable=true, displayNetwork=yorku, devices=[Ljava.lang.Object;@4c012462, display=qxl, maxVCpus=16, clientIp=, statusTime=5257430140, maxMemSlots=16}], log id: 2808a05a 2017-10-01 16:35:25,479-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler6) [485f9af] Received a memballoon Device without an address when processing VM ae56c31c-71f7-433a-b198-a1c2c846325e devices, skipping device: {device=memballoon, specParams={model=none}, type=balloon, alias=balloon0} 2017-10-01 16:35:25,479-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler6) [485f9af] Received a spice Device without an address when processing VM ae56c31c-71f7-433a-b198-a1c2c846325e devices, skipping device: {device=spice, specParams={fileTransferEnable=true, displayNetwork=yorku, copyPasteEnable=true, displayIp=130.63.94.37, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,ssmartcard,susbredir}, type=graphics, deviceId=00000030-0030-0030-0030-0000000001cd, tlsPort=5906} 2017-10-01 16:35:29,987-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler3) [568a1ff6] VM 'd66235b4-7c8d-4da2-a572-82ecc9c53a59'(www-las) moved from 'WaitForLaunch' --> 'PoweringUp' 2017-10-01 16:35:29,999-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVDSCommand] (DefaultQuartzScheduler3) [568a1ff6] START, FullListVDSCommand(HostName = virt3, FullListVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vmIds='[d66235b4-7c8d-4da2-a572-82ecc9c53a59]'}), log id: 67f7f3ff 2017-10-01 16:35:31,015-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVDSCommand] (DefaultQuartzScheduler3) [568a1ff6] FINISH, FullListVDSCommand, return: [{acpiEnable=true, emulatedMachine=pc-i440fx-rhel7.3.0, vmId=d66235b4-7c8d-4da2-a572-82ecc9c53a59, guestDiskMapping={}, transparentHugePages=true, timeOffset=0, cpuType=SandyBridge, smp=4, pauseCode=NOERR, guestNumaNodes=[Ljava.lang.Object;@22bde70e, smartcardEnable=false, custom={device_e6f104af-4245-4a6d-8c1d-ecc3120daaabdevice_783850c4-36cd-49c5-b1b7-1b94ad859083=VmDevice:{id='VmDeviceId:{deviceId='783850c4-36cd-49c5-b1b7-1b94ad859083', vmId='d66235b4-7c8d-4da2-a572-82ecc9c53a59'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=2}', managed='false', plugged='true', readOnly='false', deviceAlias='channel1', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_e6f104af-4245-4a6d-8c1d-ecc3120daaabdevice_783850c4-36cd-49c5-b1b7-1b94ad859083device_d94441b2-fa26-4878-b6d1-468c3d8cc29edevice_905471fd-92ff-4c8a-ae24-c8b018279497=VmDevice:{id='VmDeviceId:{deviceId='905471fd-92ff-4c8a-ae24-c8b018279497', vmId='d66235b4-7c8d-4da2-a572-82ecc9c53a59'}', device='ide', type='CONTROLLER', bootOrder='0', specParams='[]', address='{slot=0x01, bus=0x00, domain=0x0000, type=pci, function=0x1}', managed='false', plugged='true', readOnly='false', deviceAlias='ide', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_e6f104af-4245-4a6d-8c1d-ecc3120daaab=VmDevice:{id='VmDeviceId:{deviceId='e6f104af-4245-4a6d-8c1d-ecc3120daaab', vmId='d66235b4-7c8d-4da2-a572-82ecc9c53a59'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=1}', managed='false', plugged='true', readOnly='false', deviceAlias='channel0', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_e6f104af-4245-4a6d-8c1d-ecc3120daaabdevice_783850c4-36cd-49c5-b1b7-1b94ad859083device_d94441b2-fa26-4878-b6d1-468c3d8cc29e=VmDevice:{id='VmDeviceId:{deviceId='d94441b2-fa26-4878-b6d1-468c3d8cc29e', vmId='d66235b4-7c8d-4da2-a572-82ecc9c53a59'}', device='spicevmc', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=3}', managed='false', plugged='true', readOnly='false', deviceAlias='channel2', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}}, vmType=kvm, memSize=4096, smpCoresPerSocket=1, vmName=www-las, nice=0, status=Up, maxMemSize=16384, bootMenuEnable=false, pid=8130, smpThreadsPerCore=1, memGuaranteedSize=4096, kvmEnable=true, displayNetwork=yorku, devices=[Ljava.lang.Object;@1ad32747, display=qxl, maxVCpus=16, clientIp=, statusTime=5262746460, maxMemSlots=16}], log id: 67f7f3ff 2017-10-01 16:35:31,016-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler3) [568a1ff6] Received a memballoon Device without an address when processing VM d66235b4-7c8d-4da2-a572-82ecc9c53a59 devices, skipping device: {device=memballoon, specParams={model=none}, type=balloon, alias=balloon0} 2017-10-01 16:35:31,016-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler3) [568a1ff6] Received a spice Device without an address when processing VM d66235b4-7c8d-4da2-a572-82ecc9c53a59 devices, skipping device: {device=spice, specParams={fileTransferEnable=true, displayNetwork=yorku, copyPasteEnable=true, displayIp=130.63.94.36, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,ssmartcard,susbredir}, type=graphics, deviceId=00000032-0032-0032-0032-0000000002c8, tlsPort=5908} 2017-10-01 16:35:41,353-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler6) [5d651bf1] VM 'c64d80f9-a3bd-44f3-89c4-f1b9b509cdde'(red) moved from 'WaitForLaunch' --> 'PoweringUp' 2017-10-01 16:35:41,377-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVDSCommand] (DefaultQuartzScheduler6) [5d651bf1] START, FullListVDSCommand(HostName = virt1, FullListVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', vmIds='[c64d80f9-a3bd-44f3-89c4-f1b9b509cdde]'}), log id: 3ed5040c 2017-10-01 16:35:42,390-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVDSCommand] (DefaultQuartzScheduler6) [5d651bf1] FINISH, FullListVDSCommand, return: [{acpiEnable=true, emulatedMachine=pc-i440fx-rhel7.3.0, vmId=c64d80f9-a3bd-44f3-89c4-f1b9b509cdde, guestDiskMapping={}, transparentHugePages=true, timeOffset=0, cpuType=SandyBridge, smp=6, pauseCode=NOERR, guestNumaNodes=[Ljava.lang.Object;@3e2ce7c, smartcardEnable=false, custom={device_16dad27f-1884-4f32-8736-6bbaf8cd2fbddevice_d9f2931e-e8e0-4b83-8534-9c35daabff71device_28c1e908-e275-411a-bbeb-2920a3032e74=VmDevice:{id='VmDeviceId:{deviceId='28c1e908-e275-411a-bbeb-2920a3032e74', vmId='c64d80f9-a3bd-44f3-89c4-f1b9b509cdde'}', device='spicevmc', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=3}', managed='false', plugged='true', readOnly='false', deviceAlias='channel2', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_16dad27f-1884-4f32-8736-6bbaf8cd2fbddevice_d9f2931e-e8e0-4b83-8534-9c35daabff71=VmDevice:{id='VmDeviceId:{deviceId='d9f2931e-e8e0-4b83-8534-9c35daabff71', vmId='c64d80f9-a3bd-44f3-89c4-f1b9b509cdde'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=2}', managed='false', plugged='true', readOnly='false', deviceAlias='channel1', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_16dad27f-1884-4f32-8736-6bbaf8cd2fbddevice_d9f2931e-e8e0-4b83-8534-9c35daabff71device_28c1e908-e275-411a-bbeb-2920a3032e74device_44b628f3-6129-417b-a46d-de82d1b20d76=VmDevice:{id='VmDeviceId:{deviceId='44b628f3-6129-417b-a46d-de82d1b20d76', vmId='c64d80f9-a3bd-44f3-89c4-f1b9b509cdde'}', device='ide', type='CONTROLLER', bootOrder='0', specParams='[]', address='{slot=0x01, bus=0x00, domain=0x0000, type=pci, function=0x1}', managed='false', plugged='true', readOnly='false', deviceAlias='ide', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_16dad27f-1884-4f32-8736-6bbaf8cd2fbd=VmDevice:{id='VmDeviceId:{deviceId='16dad27f-1884-4f32-8736-6bbaf8cd2fbd', vmId='c64d80f9-a3bd-44f3-89c4-f1b9b509cdde'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=1}', managed='false', plugged='true', readOnly='false', deviceAlias='channel0', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}}, vmType=kvm, memSize=20480, smpCoresPerSocket=1, vmName=red, nice=0, status=Up, maxMemSize=81920, bootMenuEnable=false, pid=5226, smpThreadsPerCore=1, memGuaranteedSize=16384, kvmEnable=true, displayNetwork=yorku, devices=[Ljava.lang.Object;@62d89704, display=qxl, maxVCpus=16, clientIp=, statusTime=4295574570, maxMemSlots=16}], log id: 3ed5040c 2017-10-01 16:35:42,393-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler6) [5d651bf1] Received a memballoon Device without an address when processing VM c64d80f9-a3bd-44f3-89c4-f1b9b509cdde devices, skipping device: {device=memballoon, specParams={model=none}, type=balloon, alias=balloon0} 2017-10-01 16:35:42,393-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler6) [5d651bf1] Received a spice Device without an address when processing VM c64d80f9-a3bd-44f3-89c4-f1b9b509cdde devices, skipping device: {device=spice, specParams={fileTransferEnable=true, displayNetwork=yorku, copyPasteEnable=true, displayIp=130.63.94.34, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,ssmartcard,susbredir}, type=graphics, deviceId=0000001f-001f-001f-001f-000000000268, tlsPort=5903} 2017-10-01 16:35:56,984-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [3c7fbf52] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 16:35:58,271-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler5) [95a6377] VM '1c7e966f-29e8-4788-b64c-86e52f415061'(udb) moved from 'PoweringUp' --> 'Up' 2017-10-01 16:35:58,280-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [95a6377] EVENT_ID: USER_RUN_VM(32), Correlation ID: ef0a1d32-bc65-4a25-a600-995f2ad50edb, Job ID: 4ae3ca02-cd32-41f8-9eeb-4a836661152f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM udb started on Host virt1 2017-10-01 16:36:01,061-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler8) [3396df56] VM '07331c78-6e09-41b4-9b29-edda32f46351'(samba) moved from 'PoweringUp' --> 'Up' 2017-10-01 16:36:01,061-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler8) [3396df56] VM '1325cebf-834e-4571-a3e1-82beb0021ffe'(dns1) moved from 'PoweringUp' --> 'Up' 2017-10-01 16:36:01,064-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler8) [3396df56] VM 'd56363ad-9cb1-4a70-a33a-ce439b044d7d'(svn) moved from 'PoweringUp' --> 'Up' 2017-10-01 16:36:01,076-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [3396df56] EVENT_ID: USER_RUN_VM(32), Correlation ID: ef0a1d32-bc65-4a25-a600-995f2ad50edb, Job ID: 2af1dd23-b7a9-48ac-b8d1-dedea8646c2d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM svn started on Host virt3 2017-10-01 16:36:01,081-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [3396df56] EVENT_ID: USER_RUN_VM(32), Correlation ID: ef0a1d32-bc65-4a25-a600-995f2ad50edb, Job ID: 831af959-a88a-49dd-97a8-fbe3f8c74394, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM samba started on Host virt3 2017-10-01 16:36:01,085-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [3396df56] EVENT_ID: USER_RUN_VM(32), Correlation ID: ef0a1d32-bc65-4a25-a600-995f2ad50edb, Job ID: 27310415-d33c-45ba-aee5-7dc84841b6a0, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM dns1 started on Host virt3 2017-10-01 16:36:06,416-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler6) [2f6b112d] VM '1734fed5-fc97-4714-bb63-55e8f4eb6d34'(wiki) moved from 'PoweringUp' --> 'Up' 2017-10-01 16:36:06,417-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler6) [2f6b112d] VM '9ea2821e-557f-45cb-9872-1e6c5aeaf246'(rs) moved from 'PoweringUp' --> 'Up' 2017-10-01 16:36:06,417-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler6) [2f6b112d] VM 'a82b4cc3-6ef6-4a8e-97ef-1d4ea1b7e07c'(indigo) moved from 'PoweringUp' --> 'Up' 2017-10-01 16:36:06,417-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler6) [2f6b112d] VM 'aa487207-7ff4-465a-9d9b-2a103d50dc77'(ftp1) moved from 'PoweringUp' --> 'Up' 2017-10-01 16:36:06,426-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [2f6b112d] EVENT_ID: USER_RUN_VM(32), Correlation ID: ef0a1d32-bc65-4a25-a600-995f2ad50edb, Job ID: 4e144a04-558f-4e61-8910-5c153de6e0ba, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM rs started on Host virt2 2017-10-01 16:36:06,430-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [2f6b112d] EVENT_ID: USER_RUN_VM(32), Correlation ID: ef0a1d32-bc65-4a25-a600-995f2ad50edb, Job ID: 84f92459-c068-4a8c-a6a2-8e2db2344adc, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM indigo started on Host virt2 2017-10-01 16:36:06,434-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [2f6b112d] EVENT_ID: USER_RUN_VM(32), Correlation ID: ef0a1d32-bc65-4a25-a600-995f2ad50edb, Job ID: a50283b9-72f4-413d-8e17-c195edd9dd37, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM ftp1 started on Host virt2 2017-10-01 16:36:06,438-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [2f6b112d] EVENT_ID: USER_RUN_VM(32), Correlation ID: ef0a1d32-bc65-4a25-a600-995f2ad50edb, Job ID: 21061038-ed9c-408a-a715-a5e78042ce8a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM wiki started on Host virt2 2017-10-01 16:36:11,478-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler6) [41c0da80] VM 'ae56c31c-71f7-433a-b198-a1c2c846325e'(forum) moved from 'PoweringUp' --> 'Up' 2017-10-01 16:36:11,478-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler6) [41c0da80] VM '3e5a5728-8940-4fe7-b4af-11fb8e12f116'(www) moved from 'PoweringUp' --> 'Up' 2017-10-01 16:36:11,478-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler6) [41c0da80] VM 'f87ec08d-0fdf-42ba-8aac-c66c404d7aae'(webapp) moved from 'PoweringUp' --> 'Up' 2017-10-01 16:36:11,479-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler6) [41c0da80] VM '911d6883-607c-464e-b46b-8e6141bdd2de'(hdb) moved from 'PoweringUp' --> 'Up' 2017-10-01 16:36:11,479-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler6) [41c0da80] VM 'bfe9a524-2238-4bab-a91f-5dc6ed270896'(pcmode) moved from 'PoweringUp' --> 'Up' 2017-10-01 16:36:11,479-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler6) [41c0da80] VM 'c8682859-9e55-460a-9ef0-d03e4a85f50e'(dhcp1) moved from 'PoweringUp' --> 'Up' 2017-10-01 16:36:11,480-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler6) [41c0da80] VM '64734fca-4d6c-443c-82d1-abf7cf106601'(webdav) moved from 'PoweringUp' --> 'Up' 2017-10-01 16:36:11,490-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [41c0da80] EVENT_ID: USER_RUN_VM(32), Correlation ID: ef0a1d32-bc65-4a25-a600-995f2ad50edb, Job ID: 81ea0e32-f2bf-4694-8683-f2115884f51f, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM hdb started on Host virt4 2017-10-01 16:36:11,494-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [41c0da80] EVENT_ID: USER_RUN_VM(32), Correlation ID: ef0a1d32-bc65-4a25-a600-995f2ad50edb, Job ID: c826c447-fa76-4929-9d86-edd94dea51df, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM forum started on Host virt4 2017-10-01 16:36:11,498-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [41c0da80] EVENT_ID: USER_RUN_VM(32), Correlation ID: ef0a1d32-bc65-4a25-a600-995f2ad50edb, Job ID: 02e372b0-3f8b-4ec0-ba35-4698dc90ed21, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM pcmode started on Host virt4 2017-10-01 16:36:11,502-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [41c0da80] EVENT_ID: USER_RUN_VM(32), Correlation ID: ef0a1d32-bc65-4a25-a600-995f2ad50edb, Job ID: 58769063-2c71-4d80-9d7d-b94db298bbc1, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM dhcp1 started on Host virt4 2017-10-01 16:36:11,506-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [41c0da80] EVENT_ID: USER_RUN_VM(32), Correlation ID: ef0a1d32-bc65-4a25-a600-995f2ad50edb, Job ID: 12edd07c-ff48-4d5a-967b-cc767a09d706, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM webapp started on Host virt4 2017-10-01 16:36:11,510-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [41c0da80] EVENT_ID: USER_RUN_VM(32), Correlation ID: ef0a1d32-bc65-4a25-a600-995f2ad50edb, Job ID: 25019cf4-538e-47d7-902e-07157ab3b570, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM www started on Host virt4 2017-10-01 16:36:11,515-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [41c0da80] EVENT_ID: USER_RUN_VM(32), Correlation ID: ef0a1d32-bc65-4a25-a600-995f2ad50edb, Job ID: e6f0835c-d631-4459-87cd-ef560eaf3ab2, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM webdav started on Host virt4 2017-10-01 16:36:13,719-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler7) [6f2b86b9] VM 'c64d80f9-a3bd-44f3-89c4-f1b9b509cdde'(red) moved from 'PoweringUp' --> 'Up' 2017-10-01 16:36:13,721-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler7) [6f2b86b9] VM 'ae65a104-ed3f-40e0-ba92-5708cf338918'(vpn2) moved from 'PoweringUp' --> 'Up' 2017-10-01 16:36:13,729-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [6f2b86b9] EVENT_ID: USER_RUN_VM(32), Correlation ID: ef0a1d32-bc65-4a25-a600-995f2ad50edb, Job ID: 5d172da9-e7cb-4174-ad56-d1106e33b18b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM vpn2 started on Host virt1 2017-10-01 16:36:13,733-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [6f2b86b9] EVENT_ID: USER_RUN_VM(32), Correlation ID: ef0a1d32-bc65-4a25-a600-995f2ad50edb, Job ID: 5ba74392-51d3-44bd-83b8-b068d5c38233, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM red started on Host virt1 2017-10-01 16:36:16,102-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler9) [100e54bf] VM 'cc1dbe93-bb80-451e-8b3b-c9cc5e90432f'(license) moved from 'PoweringUp' --> 'Up' 2017-10-01 16:36:16,103-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler9) [100e54bf] VM 'd66235b4-7c8d-4da2-a572-82ecc9c53a59'(www-las) moved from 'PoweringUp' --> 'Up' 2017-10-01 16:36:16,103-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler9) [100e54bf] VM '3bb83ad5-dc00-445d-8708-2db18c53e0e6'(www-vhost) moved from 'PoweringUp' --> 'Up' 2017-10-01 16:36:16,104-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler9) [100e54bf] VM '59287796-c634-4f3c-94b1-2c9699986637'(vpn1) moved from 'PoweringUp' --> 'Up' 2017-10-01 16:36:16,116-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [100e54bf] EVENT_ID: USER_RUN_VM(32), Correlation ID: ef0a1d32-bc65-4a25-a600-995f2ad50edb, Job ID: 92c99462-6311-4bc7-9199-ed531cf806fa, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM license started on Host virt3 2017-10-01 16:36:16,122-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [100e54bf] EVENT_ID: USER_RUN_VM(32), Correlation ID: ef0a1d32-bc65-4a25-a600-995f2ad50edb, Job ID: a7def337-f3fb-455e-babd-44b13883cb8a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM www-las started on Host virt3 2017-10-01 16:36:16,127-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [100e54bf] EVENT_ID: USER_RUN_VM(32), Correlation ID: ef0a1d32-bc65-4a25-a600-995f2ad50edb, Job ID: b56dc0f1-596d-4d10-a246-4ca9c36dad9b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM www-vhost started on Host virt3 2017-10-01 16:36:16,138-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [100e54bf] EVENT_ID: USER_RUN_VM(32), Correlation ID: ef0a1d32-bc65-4a25-a600-995f2ad50edb, Job ID: 9224dd2d-47bb-4619-bdd4-9ef43dc27c5b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM vpn1 started on Host virt3 2017-10-01 16:36:46,489-04 INFO [org.ovirt.engine.core.bll.RunVmCommand] (default task-20) [24749604-39d1-4a75-9d4c-2da6787b045b] Lock Acquired to object 'EngineLock:{exclusiveLocks='[77766cb4-0625-4432-a16f-def5e702102a=VM]', sharedLocks=''}' 2017-10-01 16:36:46,515-04 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (default task-20) [24749604-39d1-4a75-9d4c-2da6787b045b] START, IsVmDuringInitiatingVDSCommand( IsVmDuringInitiatingVDSCommandParameters:{runAsync='true', vmId='77766cb4-0625-4432-a16f-def5e702102a'}), log id: 53d5d22b 2017-10-01 16:36:46,515-04 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (default task-20) [24749604-39d1-4a75-9d4c-2da6787b045b] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 53d5d22b 2017-10-01 16:36:46,606-04 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-6-thread-18) [24749604-39d1-4a75-9d4c-2da6787b045b] Running command: RunVmCommand internal: false. Entities affected : ID: 77766cb4-0625-4432-a16f-def5e702102a Type: VMAction group RUN_VM with role type USER 2017-10-01 16:36:46,696-04 INFO [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (org.ovirt.thread.pool-6-thread-18) [24749604-39d1-4a75-9d4c-2da6787b045b] START, UpdateVmDynamicDataVDSCommand( UpdateVmDynamicDataVDSCommandParameters:{runAsync='true', hostId='null', vmId='77766cb4-0625-4432-a16f-def5e702102a', vmDynamic='org.ovirt.engine.core.common.businessentities.VmDynamic@5b57630d'}), log id: de57cf3 2017-10-01 16:36:46,697-04 INFO [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (org.ovirt.thread.pool-6-thread-18) [24749604-39d1-4a75-9d4c-2da6787b045b] FINISH, UpdateVmDynamicDataVDSCommand, log id: de57cf3 2017-10-01 16:36:46,700-04 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-6-thread-18) [24749604-39d1-4a75-9d4c-2da6787b045b] START, CreateVmVDSCommand( CreateVmVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vmId='77766cb4-0625-4432-a16f-def5e702102a', vm='VM [labtest1]'}), log id: 641d05ba 2017-10-01 16:36:46,701-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-6-thread-18) [24749604-39d1-4a75-9d4c-2da6787b045b] START, CreateVDSCommand(HostName = virt4, CreateVmVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vmId='77766cb4-0625-4432-a16f-def5e702102a', vm='VM [labtest1]'}), log id: 3a3bc501 2017-10-01 16:36:46,705-04 INFO [org.ovirt.engine.core.vdsbroker.builder.vminfo.VmInfoBuilderImpl] (org.ovirt.thread.pool-6-thread-18) [24749604-39d1-4a75-9d4c-2da6787b045b] Bootable disk '21cfec94-32c9-4d66-a603-5439fb846fd7' set to index '0' 2017-10-01 16:36:46,719-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-6-thread-18) [24749604-39d1-4a75-9d4c-2da6787b045b] VM {memGuaranteedSize=1365, smpThreadsPerCore=1, cpuType=SandyBridge, vmId=77766cb4-0625-4432-a16f-def5e702102a, acpiEnable=true, vmType=kvm, smp=2, smpCoresPerSocket=1, emulatedMachine=pc-i440fx-rhel7.3.0, smartcardEnable=false, guestNumaNodes=[{memory=2048, cpus=0,1, nodeIndex=0}], transparentHugePages=true, displayNetwork=yorku, vmName=labtest1, maxVCpus=16, kvmEnable=true, devices=[{address={bus=0x00, domain=0x0000, function=0x0, slot=0x02, type=pci}, type=video, specParams={heads=1, vram=32768}, device=qxl, deviceId=f54f3bd7-2a59-4c25-90ed-30b40621e28a}, {type=graphics, specParams={fileTransferEnable=true, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,ssmartcard,susbredir, copyPasteEnable=true}, device=spice, deviceId=00000020-0020-0020-0020-0000000000ff}, {iface=ide, shared=false, path=, address={bus=1, controller=0, unit=0, type=drive, target=0}, readonly=true, index=2, type=disk, specParams={path=}, device=cdrom, deviceId=f2509c75-eb84-4e89-b6c3-83a26a65cc6a}, {discard=false, shared=false, address={bus=0, controller=0, unit=0, type=drive, target=0}, imageID=21cfec94-32c9-4d66-a603-5439fb846fd7, format=raw, index=0, optional=false, type=disk, deviceId=21cfec94-32c9-4d66-a603-5439fb846fd7, domainID=c8653dba-63ec-4447-872c-0ffcc6af9c0f, propagateErrors=off, iface=scsi, readonly=false, bootOrder=1, poolID=aca61eac-ed8a-484b-bbee-3645148f6405, volumeID=54094933-9b92-4559-bd6d-44448a511fac, specParams={}, device=disk}, {discard=false, shared=false, address={bus=0, controller=0, unit=1, type=drive, target=0}, imageID=c1c58ad9-b2b8-48cd-b80a-a36c063aa354, format=raw, optional=false, type=disk, deviceId=c1c58ad9-b2b8-48cd-b80a-a36c063aa354, domainID=c8653dba-63ec-4447-872c-0ffcc6af9c0f, propagateErrors=off, iface=scsi, readonly=false, poolID=aca61eac-ed8a-484b-bbee-3645148f6405, volumeID=31d0c00b-814f-4a22-affe-ced1e144e9d9, specParams={}, device=disk}, {filter=vdsm-no-mac-spoofing, nicModel=pv, address={bus=0x00, domain=0x0000, function=0x0, slot=0x03, type=pci}, type=interface, specParams={inbound={}, outbound={}}, device=bridge, linkActive=true, deviceId=aed60117-08a1-4ba3-a7fd-e1b7b57202be, macAddr=00:1a:4a:32:3d:21, network=yorku}, {filter=vdsm-no-mac-spoofing, nicModel=pv, address={bus=0x00, domain=0x0000, function=0x0, slot=0x04, type=pci}, type=interface, specParams={inbound={}, outbound={}}, device=bridge, linkActive=true, deviceId=72f26169-9052-4d52-8d95-c1c084bd4916, macAddr=00:1a:4a:32:3d:22, network=mrpriv}, {address={bus=0x00, domain=0x0000, function=0x2, slot=0x01, type=pci}, index=0, model=piix3-uhci, type=controller, specParams={}, device=usb, deviceId=449e45ea-6f29-42c4-8a7b-ac080f68cc2a}, {address={bus=0x00, domain=0x0000, function=0x0, slot=0x05, type=pci}, index=0, model=virtio-scsi, type=controller, specParams={}, device=scsi, deviceId=79ea4f85-826f-4619-9664-c6faa9ad07fc}, {address={bus=0x00, domain=0x0000, function=0x0, slot=0x06, type=pci}, type=controller, specParams={}, device=virtio-serial, deviceId=dee9b89f-5518-4349-b885-918774e6eb3e}, {model=virtio, type=rng, specParams={source=urandom}, device=virtio, deviceId=eb6d7c3a-c1ee-426b-a719-a949621808cc}], custom={device_b67a3273-fff0-47fb-b61f-32131ce376bfdevice_c0b12b3d-c2a1-444a-9747-8d4f9ba3aecddevice_e4ce4262-ceef-4972-987d-9c8481aaab38device_736b8038-d8d4-4150-bf5d-266fc7a90f54=VmDevice:{id='VmDeviceId:{deviceId='736b8038-d8d4-4150-bf5d-266fc7a90f54', vmId='77766cb4-0625-4432-a16f-def5e702102a'}', device='ide', type='CONTROLLER', bootOrder='0', specParams='[]', address='{slot=0x01, bus=0x00, domain=0x0000, type=pci, function=0x1}', managed='false', plugged='true', readOnly='false', deviceAlias='ide', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_b67a3273-fff0-47fb-b61f-32131ce376bf=VmDevice:{id='VmDeviceId:{deviceId='b67a3273-fff0-47fb-b61f-32131ce376bf', vmId='77766cb4-0625-4432-a16f-def5e702102a'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=1}', managed='false', plugged='true', readOnly='false', deviceAlias='channel0', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_b67a3273-fff0-47fb-b61f-32131ce376bfdevice_c0b12b3d-c2a1-444a-9747-8d4f9ba3aecddevice_e4ce4262-ceef-4972-987d-9c8481aaab38=VmDevice:{id='VmDeviceId:{deviceId='e4ce4262-ceef-4972-987d-9c8481aaab38', vmId='77766cb4-0625-4432-a16f-def5e702102a'}', device='spicevmc', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=3}', managed='false', plugged='true', readOnly='false', deviceAlias='channel2', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_b67a3273-fff0-47fb-b61f-32131ce376bfdevice_c0b12b3d-c2a1-444a-9747-8d4f9ba3aecd=VmDevice:{id='VmDeviceId:{deviceId='c0b12b3d-c2a1-444a-9747-8d4f9ba3aecd', vmId='77766cb4-0625-4432-a16f-def5e702102a'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=2}', managed='false', plugged='true', readOnly='false', deviceAlias='channel1', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}}, display=qxl, timeOffset=0, nice=0, maxMemSize=8192, maxMemSlots=16, bootMenuEnable=false, memSize=2048} 2017-10-01 16:36:46,780-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-6-thread-18) [24749604-39d1-4a75-9d4c-2da6787b045b] FINISH, CreateVDSCommand, log id: 3a3bc501 2017-10-01 16:36:46,783-04 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-6-thread-18) [24749604-39d1-4a75-9d4c-2da6787b045b] FINISH, CreateVmVDSCommand, return: WaitForLaunch, log id: 641d05ba 2017-10-01 16:36:46,785-04 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-6-thread-18) [24749604-39d1-4a75-9d4c-2da6787b045b] Lock freed to object 'EngineLock:{exclusiveLocks='[77766cb4-0625-4432-a16f-def5e702102a=VM]', sharedLocks=''}' 2017-10-01 16:36:46,796-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-18) [24749604-39d1-4a75-9d4c-2da6787b045b] EVENT_ID: USER_STARTED_VM(153), Correlation ID: 24749604-39d1-4a75-9d4c-2da6787b045b, Job ID: 8c7ee292-ecc8-4358-859f-ec331bf1a800, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM labtest1 was started by jas@eecs.yorku.ca-authz (Host: virt4). 2017-10-01 16:36:57,005-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [1f927871] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 16:36:57,663-04 INFO [org.ovirt.engine.core.bll.RunVmCommand] (default task-31) [61618ff4-e812-4539-8fce-28f9727ccce9] Lock Acquired to object 'EngineLock:{exclusiveLocks='[dc66c051-24b7-4cb8-99eb-db398d396ef1=VM]', sharedLocks=''}' 2017-10-01 16:36:57,683-04 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (default task-31) [61618ff4-e812-4539-8fce-28f9727ccce9] START, IsVmDuringInitiatingVDSCommand( IsVmDuringInitiatingVDSCommandParameters:{runAsync='true', vmId='dc66c051-24b7-4cb8-99eb-db398d396ef1'}), log id: 20218717 2017-10-01 16:36:57,683-04 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (default task-31) [61618ff4-e812-4539-8fce-28f9727ccce9] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 20218717 2017-10-01 16:36:57,773-04 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-6-thread-49) [61618ff4-e812-4539-8fce-28f9727ccce9] Running command: RunVmCommand internal: false. Entities affected : ID: dc66c051-24b7-4cb8-99eb-db398d396ef1 Type: VMAction group RUN_VM with role type USER 2017-10-01 16:36:57,857-04 INFO [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (org.ovirt.thread.pool-6-thread-49) [61618ff4-e812-4539-8fce-28f9727ccce9] START, UpdateVmDynamicDataVDSCommand( UpdateVmDynamicDataVDSCommandParameters:{runAsync='true', hostId='null', vmId='dc66c051-24b7-4cb8-99eb-db398d396ef1', vmDynamic='org.ovirt.engine.core.common.businessentities.VmDynamic@48f2880'}), log id: 4bc7d59d 2017-10-01 16:36:57,858-04 INFO [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (org.ovirt.thread.pool-6-thread-49) [61618ff4-e812-4539-8fce-28f9727ccce9] FINISH, UpdateVmDynamicDataVDSCommand, log id: 4bc7d59d 2017-10-01 16:36:57,860-04 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-6-thread-49) [61618ff4-e812-4539-8fce-28f9727ccce9] START, CreateVmVDSCommand( CreateVmVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vmId='dc66c051-24b7-4cb8-99eb-db398d396ef1', vm='VM [samba1]'}), log id: 2a6f4a78 2017-10-01 16:36:57,861-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-6-thread-49) [61618ff4-e812-4539-8fce-28f9727ccce9] START, CreateVDSCommand(HostName = virt4, CreateVmVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vmId='dc66c051-24b7-4cb8-99eb-db398d396ef1', vm='VM [samba1]'}), log id: 669123df 2017-10-01 16:36:57,864-04 INFO [org.ovirt.engine.core.vdsbroker.builder.vminfo.VmInfoBuilderImpl] (org.ovirt.thread.pool-6-thread-49) [61618ff4-e812-4539-8fce-28f9727ccce9] Bootable disk '7c7752b5-dba1-4469-88f0-0067d2042cb7' set to index '0' 2017-10-01 16:36:57,876-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-6-thread-49) [61618ff4-e812-4539-8fce-28f9727ccce9] VM {memGuaranteedSize=4096, smpThreadsPerCore=1, cpuType=SandyBridge, vmId=dc66c051-24b7-4cb8-99eb-db398d396ef1, acpiEnable=true, vmType=kvm, smp=4, smpCoresPerSocket=1, emulatedMachine=pc-i440fx-rhel7.3.0, smartcardEnable=false, guestNumaNodes=[{memory=4096, cpus=0,1,2,3, nodeIndex=0}], transparentHugePages=true, displayNetwork=yorku, vmName=samba1, maxVCpus=16, kvmEnable=true, devices=[{address={bus=0x00, domain=0x0000, function=0x0, slot=0x02, type=pci}, type=video, specParams={vgamem=16384, heads=1, vram=8192, ram=65536}, device=qxl, deviceId=cc2abe93-52e8-4ac1-83c3-ebeeac374f04}, {type=graphics, specParams={fileTransferEnable=true, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,ssmartcard,susbredir, copyPasteEnable=true}, device=spice, deviceId=00000033-0033-0033-0033-00000000028f}, {iface=ide, shared=false, path=, address={bus=1, controller=0, unit=0, type=drive, target=0}, readonly=true, index=2, type=disk, specParams={path=}, device=cdrom, deviceId=de97a601-aaf0-426e-9c4e-50af896cdacf}, {discard=false, shared=false, address={bus=0, controller=0, unit=0, type=drive, target=0}, imageID=7c7752b5-dba1-4469-88f0-0067d2042cb7, format=raw, index=0, optional=false, type=disk, deviceId=7c7752b5-dba1-4469-88f0-0067d2042cb7, domainID=c8653dba-63ec-4447-872c-0ffcc6af9c0f, propagateErrors=off, iface=scsi, readonly=false, bootOrder=1, poolID=aca61eac-ed8a-484b-bbee-3645148f6405, volumeID=85f2a71b-08bb-48da-82de-5f168218e90c, specParams={}, device=disk}, {discard=false, shared=false, address={bus=0, controller=0, unit=1, type=drive, target=0}, imageID=5dbe0c41-49ea-4e16-a107-57b313d4ebd4, format=raw, optional=false, type=disk, deviceId=5dbe0c41-49ea-4e16-a107-57b313d4ebd4, domainID=c8653dba-63ec-4447-872c-0ffcc6af9c0f, propagateErrors=off, iface=scsi, readonly=false, poolID=aca61eac-ed8a-484b-bbee-3645148f6405, volumeID=0d728285-b51e-49a1-be2e-6b39e0edaab6, specParams={}, device=disk}, {filter=vdsm-no-mac-spoofing, nicModel=pv, address={bus=0x00, domain=0x0000, function=0x0, slot=0x03, type=pci}, type=interface, specParams={inbound={}, outbound={}}, device=bridge, linkActive=true, deviceId=63fbd065-7eb7-41e8-be35-736abc0f0a98, macAddr=00:1a:4a:32:3d:3a, network=yorku}, {filter=vdsm-no-mac-spoofing, nicModel=pv, address={bus=0x00, domain=0x0000, function=0x0, slot=0x04, type=pci}, type=interface, specParams={inbound={}, outbound={}}, device=bridge, linkActive=true, deviceId=02a4ca2a-b102-44a5-8e80-bbdd7977630e, macAddr=00:1a:4a:32:3d:3b, network=mrpriv}, {address={bus=0x00, domain=0x0000, function=0x2, slot=0x01, type=pci}, index=0, model=piix3-uhci, type=controller, specParams={}, device=usb, deviceId=94c4246d-bfd3-4b2b-9872-a10e2bcf5e77}, {address={bus=0x00, domain=0x0000, function=0x0, slot=0x05, type=pci}, index=0, model=virtio-scsi, type=controller, specParams={}, device=scsi, deviceId=089c1cd8-7856-4631-bd7c-08ad86a9647f}, {address={bus=0x00, domain=0x0000, function=0x0, slot=0x06, type=pci}, type=controller, specParams={}, device=virtio-serial, deviceId=81c774f7-711d-4cb5-9621-c00b24c34d0c}, {model=virtio, type=rng, specParams={source=urandom}, device=virtio, deviceId=7caa77ab-c064-4ef0-8f86-c689974e696a}], custom={device_ad5895f2-19d6-4fd5-822b-fa0f07f52388device_7978e7b9-423b-428a-9c7b-2ff33ff426b3device_103214d1-49cd-4f6a-b1f2-944060b1acbb=VmDevice:{id='VmDeviceId:{deviceId='103214d1-49cd-4f6a-b1f2-944060b1acbb', vmId='dc66c051-24b7-4cb8-99eb-db398d396ef1'}', device='spicevmc', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=3}', managed='false', plugged='true', readOnly='false', deviceAlias='channel2', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_ad5895f2-19d6-4fd5-822b-fa0f07f52388device_7978e7b9-423b-428a-9c7b-2ff33ff426b3=VmDevice:{id='VmDeviceId:{deviceId='7978e7b9-423b-428a-9c7b-2ff33ff426b3', vmId='dc66c051-24b7-4cb8-99eb-db398d396ef1'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=2}', managed='false', plugged='true', readOnly='false', deviceAlias='channel1', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_ad5895f2-19d6-4fd5-822b-fa0f07f52388=VmDevice:{id='VmDeviceId:{deviceId='ad5895f2-19d6-4fd5-822b-fa0f07f52388', vmId='dc66c051-24b7-4cb8-99eb-db398d396ef1'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=1}', managed='false', plugged='true', readOnly='false', deviceAlias='channel0', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_ad5895f2-19d6-4fd5-822b-fa0f07f52388device_7978e7b9-423b-428a-9c7b-2ff33ff426b3device_103214d1-49cd-4f6a-b1f2-944060b1acbbdevice_85ac2656-993f-41c0-8130-19e0444d2895=VmDevice:{id='VmDeviceId:{deviceId='85ac2656-993f-41c0-8130-19e0444d2895', vmId='dc66c051-24b7-4cb8-99eb-db398d396ef1'}', device='ide', type='CONTROLLER', bootOrder='0', specParams='[]', address='{slot=0x01, bus=0x00, domain=0x0000, type=pci, function=0x1}', managed='false', plugged='true', readOnly='false', deviceAlias='ide', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}}, display=qxl, timeOffset=0, nice=0, maxMemSize=16384, maxMemSlots=16, bootMenuEnable=false, memSize=4096} 2017-10-01 16:36:58,561-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-6-thread-49) [61618ff4-e812-4539-8fce-28f9727ccce9] FINISH, CreateVDSCommand, log id: 669123df 2017-10-01 16:36:58,564-04 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-6-thread-49) [61618ff4-e812-4539-8fce-28f9727ccce9] FINISH, CreateVmVDSCommand, return: WaitForLaunch, log id: 2a6f4a78 2017-10-01 16:36:58,565-04 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-6-thread-49) [61618ff4-e812-4539-8fce-28f9727ccce9] Lock freed to object 'EngineLock:{exclusiveLocks='[dc66c051-24b7-4cb8-99eb-db398d396ef1=VM]', sharedLocks=''}' 2017-10-01 16:36:58,573-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-49) [61618ff4-e812-4539-8fce-28f9727ccce9] EVENT_ID: USER_STARTED_VM(153), Correlation ID: 61618ff4-e812-4539-8fce-28f9727ccce9, Job ID: 43c55b0d-d094-4a9f-9a56-6f56f06ea094, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM samba1 was started by jas@eecs.yorku.ca-authz (Host: virt4). 2017-10-01 16:36:58,574-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmsStatisticsFetcher] (DefaultQuartzScheduler9) [79c49f98] Fetched 10 VMs from VDS '5ae29ed2-9c0f-4754-8df3-b43d14726846' 2017-10-01 16:36:58,574-04 WARN [org.ovirt.engine.core.vdsbroker.monitoring.VmsMonitoring] (DefaultQuartzScheduler9) [79c49f98] skipping VM 'dc66c051-24b7-4cb8-99eb-db398d396ef1' from this monitoring cycle - the VM data has changed since fetching the data 2017-10-01 16:36:58,576-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler9) [79c49f98] VM '77766cb4-0625-4432-a16f-def5e702102a'(labtest1) moved from 'WaitForLaunch' --> 'PoweringUp' 2017-10-01 16:36:58,593-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVDSCommand] (DefaultQuartzScheduler9) [79c49f98] START, FullListVDSCommand(HostName = virt4, FullListVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vmIds='[dc66c051-24b7-4cb8-99eb-db398d396ef1, 77766cb4-0625-4432-a16f-def5e702102a]'}), log id: 4422ab50 2017-10-01 16:36:58,615-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVDSCommand] (DefaultQuartzScheduler9) [79c49f98] FINISH, FullListVDSCommand, return: [{acpiEnable=true, emulatedMachine=pc-i440fx-rhel7.3.0, vmId=77766cb4-0625-4432-a16f-def5e702102a, guestDiskMapping={}, transparentHugePages=true, timeOffset=0, cpuType=SandyBridge, smp=2, pauseCode=NOERR, guestNumaNodes=[Ljava.lang.Object;@5c9af922, smartcardEnable=false, custom={device_b67a3273-fff0-47fb-b61f-32131ce376bfdevice_c0b12b3d-c2a1-444a-9747-8d4f9ba3aecddevice_e4ce4262-ceef-4972-987d-9c8481aaab38=VmDevice:{id='VmDeviceId:{deviceId='e4ce4262-ceef-4972-987d-9c8481aaab38', vmId='77766cb4-0625-4432-a16f-def5e702102a'}', device='spicevmc', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=3}', managed='false', plugged='true', readOnly='false', deviceAlias='channel2', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_b67a3273-fff0-47fb-b61f-32131ce376bfdevice_c0b12b3d-c2a1-444a-9747-8d4f9ba3aecd=VmDevice:{id='VmDeviceId:{deviceId='c0b12b3d-c2a1-444a-9747-8d4f9ba3aecd', vmId='77766cb4-0625-4432-a16f-def5e702102a'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=2}', managed='false', plugged='true', readOnly='false', deviceAlias='channel1', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_b67a3273-fff0-47fb-b61f-32131ce376bfdevice_c0b12b3d-c2a1-444a-9747-8d4f9ba3aecddevice_e4ce4262-ceef-4972-987d-9c8481aaab38device_736b8038-d8d4-4150-bf5d-266fc7a90f54=VmDevice:{id='VmDeviceId:{deviceId='736b8038-d8d4-4150-bf5d-266fc7a90f54', vmId='77766cb4-0625-4432-a16f-def5e702102a'}', device='ide', type='CONTROLLER', bootOrder='0', specParams='[]', address='{slot=0x01, bus=0x00, domain=0x0000, type=pci, function=0x1}', managed='false', plugged='true', readOnly='false', deviceAlias='ide', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_b67a3273-fff0-47fb-b61f-32131ce376bf=VmDevice:{id='VmDeviceId:{deviceId='b67a3273-fff0-47fb-b61f-32131ce376bf', vmId='77766cb4-0625-4432-a16f-def5e702102a'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=1}', managed='false', plugged='true', readOnly='false', deviceAlias='channel0', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}}, vmType=kvm, memSize=2048, smpCoresPerSocket=1, vmName=labtest1, nice=0, status=Up, maxMemSize=8192, bootMenuEnable=false, pid=109397, smpThreadsPerCore=1, memGuaranteedSize=1365, kvmEnable=true, displayNetwork=yorku, devices=[Ljava.lang.Object;@1050fb94, display=qxl, maxVCpus=16, clientIp=, statusTime=5257523270, maxMemSlots=16}, {acpiEnable=true, emulatedMachine=pc-i440fx-rhel7.3.0, vmId=dc66c051-24b7-4cb8-99eb-db398d396ef1, guestDiskMapping={}, transparentHugePages=true, timeOffset=0, cpuType=SandyBridge, smp=4, guestNumaNodes=[Ljava.lang.Object;@4ea59d48, smartcardEnable=false, custom={device_ad5895f2-19d6-4fd5-822b-fa0f07f52388=VmDevice:{id='VmDeviceId:{deviceId='ad5895f2-19d6-4fd5-822b-fa0f07f52388', vmId='dc66c051-24b7-4cb8-99eb-db398d396ef1'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=1}', managed='false', plugged='true', readOnly='false', deviceAlias='channel0', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_ad5895f2-19d6-4fd5-822b-fa0f07f52388device_7978e7b9-423b-428a-9c7b-2ff33ff426b3=VmDevice:{id='VmDeviceId:{deviceId='7978e7b9-423b-428a-9c7b-2ff33ff426b3', vmId='dc66c051-24b7-4cb8-99eb-db398d396ef1'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=2}', managed='false', plugged='true', readOnly='false', deviceAlias='channel1', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_ad5895f2-19d6-4fd5-822b-fa0f07f52388device_7978e7b9-423b-428a-9c7b-2ff33ff426b3device_103214d1-49cd-4f6a-b1f2-944060b1acbbdevice_85ac2656-993f-41c0-8130-19e0444d2895=VmDevice:{id='VmDeviceId:{deviceId='85ac2656-993f-41c0-8130-19e0444d2895', vmId='dc66c051-24b7-4cb8-99eb-db398d396ef1'}', device='ide', type='CONTROLLER', bootOrder='0', specParams='[]', address='{slot=0x01, bus=0x00, domain=0x0000, type=pci, function=0x1}', managed='false', plugged='true', readOnly='false', deviceAlias='ide', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_ad5895f2-19d6-4fd5-822b-fa0f07f52388device_7978e7b9-423b-428a-9c7b-2ff33ff426b3device_103214d1-49cd-4f6a-b1f2-944060b1acbb=VmDevice:{id='VmDeviceId:{deviceId='103214d1-49cd-4f6a-b1f2-944060b1acbb', vmId='dc66c051-24b7-4cb8-99eb-db398d396ef1'}', device='spicevmc', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=3}', managed='false', plugged='true', readOnly='false', deviceAlias='channel2', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}}, vmType=kvm, memSize=4096, smpCoresPerSocket=1, vmName=samba1, nice=0, status=WaitForLaunch, maxMemSize=16384, bootMenuEnable=false, pid=0, smpThreadsPerCore=1, memGuaranteedSize=4096, kvmEnable=true, displayNetwork=yorku, devices=[Ljava.lang.Object;@1f0cc7e7, display=qxl, maxVCpus=16, clientIp=, statusTime=5257523270, maxMemSlots=16}], log id: 4422ab50 2017-10-01 16:36:58,618-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler9) [79c49f98] Received a memballoon Device without an address when processing VM 77766cb4-0625-4432-a16f-def5e702102a devices, skipping device: {device=memballoon, specParams={model=none}, type=balloon, alias=balloon0} 2017-10-01 16:36:58,618-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler9) [79c49f98] Received a spice Device without an address when processing VM 77766cb4-0625-4432-a16f-def5e702102a devices, skipping device: {device=spice, specParams={fileTransferEnable=true, displayNetwork=yorku, copyPasteEnable=true, displayIp=130.63.94.37, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,ssmartcard,susbredir}, type=graphics, deviceId=00000020-0020-0020-0020-0000000000ff, tlsPort=5908} 2017-10-01 16:36:58,619-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler9) [79c49f98] Received a memballoon Device without an address when processing VM dc66c051-24b7-4cb8-99eb-db398d396ef1 devices, skipping device: {device=memballoon, specParams={model=none}, type=balloon} 2017-10-01 16:36:58,619-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler9) [79c49f98] Received a virtio Device without an address when processing VM dc66c051-24b7-4cb8-99eb-db398d396ef1 devices, skipping device: {device=virtio, specParams={source=urandom}, model=virtio, type=rng, deviceId=7caa77ab-c064-4ef0-8f86-c689974e696a} 2017-10-01 16:36:58,619-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler9) [79c49f98] Received a spice Device without an address when processing VM dc66c051-24b7-4cb8-99eb-db398d396ef1 devices, skipping device: {device=spice, specParams={fileTransferEnable=true, displayNetwork=yorku, copyPasteEnable=true, displayIp=130.63.94.37, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,ssmartcard,susbredir}, type=graphics, deviceId=00000033-0033-0033-0033-00000000028f} 2017-10-01 16:37:13,439-04 INFO [org.ovirt.engine.core.bll.RunVmCommand] (default task-56) [c91b7cff-5a83-4833-822d-c3f0de418e5a] Lock Acquired to object 'EngineLock:{exclusiveLocks='[bcd5776e-978b-4723-933d-70a80322a796=VM]', sharedLocks=''}' 2017-10-01 16:37:13,459-04 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (default task-56) [c91b7cff-5a83-4833-822d-c3f0de418e5a] START, IsVmDuringInitiatingVDSCommand( IsVmDuringInitiatingVDSCommandParameters:{runAsync='true', vmId='bcd5776e-978b-4723-933d-70a80322a796'}), log id: 2d07fca2 2017-10-01 16:37:13,459-04 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (default task-56) [c91b7cff-5a83-4833-822d-c3f0de418e5a] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 2d07fca2 2017-10-01 16:37:13,553-04 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-6-thread-50) [c91b7cff-5a83-4833-822d-c3f0de418e5a] Running command: RunVmCommand internal: false. Entities affected : ID: bcd5776e-978b-4723-933d-70a80322a796 Type: VMAction group RUN_VM with role type USER 2017-10-01 16:37:13,637-04 INFO [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (org.ovirt.thread.pool-6-thread-50) [c91b7cff-5a83-4833-822d-c3f0de418e5a] START, UpdateVmDynamicDataVDSCommand( UpdateVmDynamicDataVDSCommandParameters:{runAsync='true', hostId='null', vmId='bcd5776e-978b-4723-933d-70a80322a796', vmDynamic='org.ovirt.engine.core.common.businessentities.VmDynamic@3806584f'}), log id: 3b6acc80 2017-10-01 16:37:13,639-04 INFO [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (org.ovirt.thread.pool-6-thread-50) [c91b7cff-5a83-4833-822d-c3f0de418e5a] FINISH, UpdateVmDynamicDataVDSCommand, log id: 3b6acc80 2017-10-01 16:37:13,640-04 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-6-thread-50) [c91b7cff-5a83-4833-822d-c3f0de418e5a] START, CreateVmVDSCommand( CreateVmVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vmId='bcd5776e-978b-4723-933d-70a80322a796', vm='VM [cape]'}), log id: 4f13039d 2017-10-01 16:37:13,642-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-6-thread-50) [c91b7cff-5a83-4833-822d-c3f0de418e5a] START, CreateVDSCommand(HostName = virt4, CreateVmVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vmId='bcd5776e-978b-4723-933d-70a80322a796', vm='VM [cape]'}), log id: 484d366d 2017-10-01 16:37:13,644-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler7) [7629f87c] VM 'dc66c051-24b7-4cb8-99eb-db398d396ef1'(samba1) moved from 'WaitForLaunch' --> 'PoweringUp' 2017-10-01 16:37:13,645-04 INFO [org.ovirt.engine.core.vdsbroker.builder.vminfo.VmInfoBuilderImpl] (org.ovirt.thread.pool-6-thread-50) [c91b7cff-5a83-4833-822d-c3f0de418e5a] Bootable disk '38cf7eec-9c32-46ed-a7fc-2148d372bbd7' set to index '0' 2017-10-01 16:37:13,658-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-6-thread-50) [c91b7cff-5a83-4833-822d-c3f0de418e5a] VM {memGuaranteedSize=5461, smpThreadsPerCore=1, cpuType=SandyBridge, vmId=bcd5776e-978b-4723-933d-70a80322a796, acpiEnable=true, vmType=kvm, smp=2, smpCoresPerSocket=1, emulatedMachine=pc-i440fx-rhel7.3.0, smartcardEnable=false, guestNumaNodes=[{memory=8192, cpus=0,1, nodeIndex=0}], transparentHugePages=true, displayNetwork=yorku, vmName=cape, maxVCpus=16, kvmEnable=true, devices=[{address={bus=0x00, domain=0x0000, function=0x7, slot=0x07, type=pci}, index=0, model=ich9-ehci1, type=controller, specParams={}, device=usb, deviceId=638b6f96-8b73-4943-a394-f0302056f822}, {address={bus=0x00, domain=0x0000, function=0x0, slot=0x02, type=pci}, type=video, specParams={vgamem=16384, heads=1, vram=8192, ram=65536}, device=qxl, deviceId=93bde27e-acdc-4748-baa0-3a319a7f8404}, {type=graphics, specParams={fileTransferEnable=true, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,ssmartcard,susbredir, copyPasteEnable=true}, device=spice, deviceId=a09ffaa5-fede-473c-a53a-986424dce8e4}, {iface=ide, shared=false, path=, address={bus=1, controller=0, unit=0, type=drive, target=0}, readonly=true, index=2, type=disk, specParams={path=}, device=cdrom, deviceId=7b4fa741-320b-41a0-9889-60b22a369ff7}, {discard=false, shared=false, address={bus=0, controller=0, unit=0, type=drive, target=0}, imageID=38cf7eec-9c32-46ed-a7fc-2148d372bbd7, format=raw, index=0, optional=false, type=disk, deviceId=38cf7eec-9c32-46ed-a7fc-2148d372bbd7, domainID=c8653dba-63ec-4447-872c-0ffcc6af9c0f, propagateErrors=off, iface=scsi, readonly=false, bootOrder=1, poolID=aca61eac-ed8a-484b-bbee-3645148f6405, volumeID=9803b26a-9b4e-4489-b390-90cc93720750, specParams={}, device=disk}, {filter=vdsm-no-mac-spoofing, nicModel=pv, address={bus=0x00, domain=0x0000, function=0x0, slot=0x03, type=pci}, type=interface, specParams={inbound={}, outbound={}}, device=bridge, linkActive=true, deviceId=13d39e4a-98b7-4194-a558-5fd3676bafc3, macAddr=00:1a:4a:32:3d:44, network=yorku}, {address={bus=0x00, domain=0x0000, function=0x2, slot=0x07, type=pci}, index=0, model=ich9-uhci3, type=controller, specParams={}, device=usb, deviceId=cdfc7994-7c66-4f31-9a07-8ab8afba4f87}, {address={bus=0x00, function=0x0, domain=0x0000, slot=0x07, type=pci, multifunction=on}, index=0, model=ich9-uhci1, type=controller, specParams={}, device=usb, deviceId=06f07623-7b12-4018-9c14-7ac6c5d8e09d}, {address={bus=0x00, domain=0x0000, function=0x1, slot=0x07, type=pci}, index=0, model=ich9-uhci2, type=controller, specParams={}, device=usb, deviceId=709adcf8-b986-4652-966c-1428af883306}, {bus=usb, type=redir, specParams={}, device=spicevmc, deviceId=8241e2e6-7340-40c8-a4f1-92e4ccff440c}, {bus=usb, type=redir, specParams={}, device=spicevmc, deviceId=9a03ca98-987d-4309-bd8b-20fd230c9057}, {bus=usb, type=redir, specParams={}, device=spicevmc, deviceId=f60084fb-a381-49f3-8006-d45bc17e0312}, {bus=usb, type=redir, specParams={}, device=spicevmc, deviceId=48522fd4-dfea-48f7-83a4-9c574b884f77}, {address={bus=0x00, function=0x0, domain=0x0000, slot=0x06, type=pci}, type=balloon, specParams={model=virtio}, device=memballoon, deviceId=8907fa22-8744-47e4-8016-5c06c9d5a6c9}, {address={bus=0x00, domain=0x0000, function=0x0, slot=0x04, type=pci}, index=0, model=virtio-scsi, type=controller, specParams={}, device=scsi, deviceId=3528781c-199f-4aef-bb48-4f65ce06bf7d}, {address={bus=0x00, domain=0x0000, function=0x0, slot=0x05, type=pci}, type=controller, specParams={}, device=virtio-serial, deviceId=98cc54b9-1c8b-4eb8-8ff4-c67a4108bc9a}, {model=virtio, type=rng, specParams={source=urandom}, device=virtio, deviceId=c28508ff-c47c-4143-a63d-7443b43753df}], custom={device_2c3e15c9-1aba-4575-8d80-08f6b5a4e972device_002231b3-f832-41a0-920f-ffbed0a1e4cbdevice_bd98dcd7-8357-48c7-9c30-d5d175a5404adevice_f25c837c-fdf1-47c5-88ce-53e75efa9f13device_fff4e79f-0f92-419c-b778-961e316b2d64device_211c4b7b-605c-481c-8931-89cd68cf9f42=VmDevice:{id='VmDeviceId:{deviceId='211c4b7b-605c-481c-8931-89cd68cf9f42', vmId='bcd5776e-978b-4723-933d-70a80322a796'}', device='spicevmc', type='REDIRDEV', bootOrder='0', specParams='[]', address='{bus=0, type=usb, port=2}', managed='false', plugged='true', readOnly='false', deviceAlias='redir1', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_2c3e15c9-1aba-4575-8d80-08f6b5a4e972device_002231b3-f832-41a0-920f-ffbed0a1e4cbdevice_bd98dcd7-8357-48c7-9c30-d5d175a5404adevice_f25c837c-fdf1-47c5-88ce-53e75efa9f13device_fff4e79f-0f92-419c-b778-961e316b2d64device_211c4b7b-605c-481c-8931-89cd68cf9f42device_e80fb0df-9d7d-4d57-8031-b86198e57969device_a44c133d-b548-4dd3-be67-0fcbc1d244c4=VmDevice:{id='VmDeviceId:{deviceId='a44c133d-b548-4dd3-be67-0fcbc1d244c4', vmId='bcd5776e-978b-4723-933d-70a80322a796'}', device='spicevmc', type='REDIRDEV', bootOrder='0', specParams='[]', address='{bus=0, type=usb, port=4}', managed='false', plugged='true', readOnly='false', deviceAlias='redir3', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_2c3e15c9-1aba-4575-8d80-08f6b5a4e972device_002231b3-f832-41a0-920f-ffbed0a1e4cbdevice_bd98dcd7-8357-48c7-9c30-d5d175a5404a=VmDevice:{id='VmDeviceId:{deviceId='bd98dcd7-8357-48c7-9c30-d5d175a5404a', vmId='bcd5776e-978b-4723-933d-70a80322a796'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=2}', managed='false', plugged='true', readOnly='false', deviceAlias='channel1', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_2c3e15c9-1aba-4575-8d80-08f6b5a4e972device_002231b3-f832-41a0-920f-ffbed0a1e4cbdevice_bd98dcd7-8357-48c7-9c30-d5d175a5404adevice_f25c837c-fdf1-47c5-88ce-53e75efa9f13device_fff4e79f-0f92-419c-b778-961e316b2d64=VmDevice:{id='VmDeviceId:{deviceId='fff4e79f-0f92-419c-b778-961e316b2d64', vmId='bcd5776e-978b-4723-933d-70a80322a796'}', device='spicevmc', type='REDIRDEV', bootOrder='0', specParams='[]', address='{bus=0, type=usb, port=1}', managed='false', plugged='true', readOnly='false', deviceAlias='redir0', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_2c3e15c9-1aba-4575-8d80-08f6b5a4e972device_002231b3-f832-41a0-920f-ffbed0a1e4cbdevice_bd98dcd7-8357-48c7-9c30-d5d175a5404adevice_f25c837c-fdf1-47c5-88ce-53e75efa9f13device_fff4e79f-0f92-419c-b778-961e316b2d64device_211c4b7b-605c-481c-8931-89cd68cf9f42device_e80fb0df-9d7d-4d57-8031-b86198e57969=VmDevice:{id='VmDeviceId:{deviceId='e80fb0df-9d7d-4d57-8031-b86198e57969', vmId='bcd5776e-978b-4723-933d-70a80322a796'}', device='spicevmc', type='REDIRDEV', bootOrder='0', specParams='[]', address='{bus=0, type=usb, port=3}', managed='false', plugged='true', readOnly='false', deviceAlias='redir2', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_2c3e15c9-1aba-4575-8d80-08f6b5a4e972=VmDevice:{id='VmDeviceId:{deviceId='2c3e15c9-1aba-4575-8d80-08f6b5a4e972', vmId='bcd5776e-978b-4723-933d-70a80322a796'}', device='ide', type='CONTROLLER', bootOrder='0', specParams='[]', address='{slot=0x01, bus=0x00, domain=0x0000, type=pci, function=0x1}', managed='false', plugged='true', readOnly='false', deviceAlias='ide', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_2c3e15c9-1aba-4575-8d80-08f6b5a4e972device_002231b3-f832-41a0-920f-ffbed0a1e4cbdevice_bd98dcd7-8357-48c7-9c30-d5d175a5404adevice_f25c837c-fdf1-47c5-88ce-53e75efa9f13=VmDevice:{id='VmDeviceId:{deviceId='f25c837c-fdf1-47c5-88ce-53e75efa9f13', vmId='bcd5776e-978b-4723-933d-70a80322a796'}', device='spicevmc', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=3}', managed='false', plugged='true', readOnly='false', deviceAlias='channel2', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_2c3e15c9-1aba-4575-8d80-08f6b5a4e972device_002231b3-f832-41a0-920f-ffbed0a1e4cb=VmDevice:{id='VmDeviceId:{deviceId='002231b3-f832-41a0-920f-ffbed0a1e4cb', vmId='bcd5776e-978b-4723-933d-70a80322a796'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=1}', managed='false', plugged='true', readOnly='false', deviceAlias='channel0', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}}, display=qxl, hypervEnable=true, timeOffset=-14400, nice=0, maxMemSlots=16, bootMenuEnable=false, memSize=8192} 2017-10-01 16:37:13,659-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVDSCommand] (DefaultQuartzScheduler7) [7629f87c] START, FullListVDSCommand(HostName = virt4, FullListVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vmIds='[dc66c051-24b7-4cb8-99eb-db398d396ef1]'}), log id: 7bd534fc 2017-10-01 16:37:13,674-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-6-thread-50) [c91b7cff-5a83-4833-822d-c3f0de418e5a] FINISH, CreateVDSCommand, log id: 484d366d 2017-10-01 16:37:13,676-04 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-6-thread-50) [c91b7cff-5a83-4833-822d-c3f0de418e5a] FINISH, CreateVmVDSCommand, return: WaitForLaunch, log id: 4f13039d 2017-10-01 16:37:13,678-04 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-6-thread-50) [c91b7cff-5a83-4833-822d-c3f0de418e5a] Lock freed to object 'EngineLock:{exclusiveLocks='[bcd5776e-978b-4723-933d-70a80322a796=VM]', sharedLocks=''}' 2017-10-01 16:37:13,682-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-50) [c91b7cff-5a83-4833-822d-c3f0de418e5a] EVENT_ID: USER_STARTED_VM(153), Correlation ID: c91b7cff-5a83-4833-822d-c3f0de418e5a, Job ID: d4f6261c-5ada-4f21-996e-263cb267c54e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM cape was started by jas@eecs.yorku.ca-authz (Host: virt4). 2017-10-01 16:37:13,684-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVDSCommand] (DefaultQuartzScheduler7) [7629f87c] FINISH, FullListVDSCommand, return: [{acpiEnable=true, emulatedMachine=pc-i440fx-rhel7.3.0, vmId=dc66c051-24b7-4cb8-99eb-db398d396ef1, guestDiskMapping={}, transparentHugePages=true, timeOffset=0, cpuType=SandyBridge, smp=4, pauseCode=NOERR, guestNumaNodes=[Ljava.lang.Object;@3a6c65bd, smartcardEnable=false, custom={device_ad5895f2-19d6-4fd5-822b-fa0f07f52388=VmDevice:{id='VmDeviceId:{deviceId='ad5895f2-19d6-4fd5-822b-fa0f07f52388', vmId='dc66c051-24b7-4cb8-99eb-db398d396ef1'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=1}', managed='false', plugged='true', readOnly='false', deviceAlias='channel0', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_ad5895f2-19d6-4fd5-822b-fa0f07f52388device_7978e7b9-423b-428a-9c7b-2ff33ff426b3=VmDevice:{id='VmDeviceId:{deviceId='7978e7b9-423b-428a-9c7b-2ff33ff426b3', vmId='dc66c051-24b7-4cb8-99eb-db398d396ef1'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=2}', managed='false', plugged='true', readOnly='false', deviceAlias='channel1', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_ad5895f2-19d6-4fd5-822b-fa0f07f52388device_7978e7b9-423b-428a-9c7b-2ff33ff426b3device_103214d1-49cd-4f6a-b1f2-944060b1acbbdevice_85ac2656-993f-41c0-8130-19e0444d2895=VmDevice:{id='VmDeviceId:{deviceId='85ac2656-993f-41c0-8130-19e0444d2895', vmId='dc66c051-24b7-4cb8-99eb-db398d396ef1'}', device='ide', type='CONTROLLER', bootOrder='0', specParams='[]', address='{slot=0x01, bus=0x00, domain=0x0000, type=pci, function=0x1}', managed='false', plugged='true', readOnly='false', deviceAlias='ide', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_ad5895f2-19d6-4fd5-822b-fa0f07f52388device_7978e7b9-423b-428a-9c7b-2ff33ff426b3device_103214d1-49cd-4f6a-b1f2-944060b1acbb=VmDevice:{id='VmDeviceId:{deviceId='103214d1-49cd-4f6a-b1f2-944060b1acbb', vmId='dc66c051-24b7-4cb8-99eb-db398d396ef1'}', device='spicevmc', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=3}', managed='false', plugged='true', readOnly='false', deviceAlias='channel2', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}}, vmType=kvm, memSize=4096, smpCoresPerSocket=1, vmName=samba1, nice=0, status=Up, maxMemSize=16384, bootMenuEnable=false, pid=109617, smpThreadsPerCore=1, memGuaranteedSize=4096, kvmEnable=true, displayNetwork=yorku, devices=[Ljava.lang.Object;@57b8569, display=qxl, maxVCpus=16, clientIp=, statusTime=5257538350, maxMemSlots=16}], log id: 7bd534fc 2017-10-01 16:37:13,685-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler7) [7629f87c] Received a memballoon Device without an address when processing VM dc66c051-24b7-4cb8-99eb-db398d396ef1 devices, skipping device: {device=memballoon, specParams={model=none}, type=balloon, alias=balloon0} 2017-10-01 16:37:13,685-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler7) [7629f87c] Received a spice Device without an address when processing VM dc66c051-24b7-4cb8-99eb-db398d396ef1 devices, skipping device: {device=spice, specParams={fileTransferEnable=true, displayNetwork=yorku, copyPasteEnable=true, displayIp=130.63.94.37, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,ssmartcard,susbredir}, type=graphics, deviceId=00000033-0033-0033-0033-00000000028f, tlsPort=5909} 2017-10-01 16:37:28,710-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmsStatisticsFetcher] (DefaultQuartzScheduler8) [4c8fa77d] Fetched 11 VMs from VDS '5ae29ed2-9c0f-4754-8df3-b43d14726846' 2017-10-01 16:37:28,710-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler8) [4c8fa77d] VM 'bcd5776e-978b-4723-933d-70a80322a796'(cape) moved from 'WaitForLaunch' --> 'PoweringUp' 2017-10-01 16:37:28,724-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVDSCommand] (DefaultQuartzScheduler8) [4c8fa77d] START, FullListVDSCommand(HostName = virt4, FullListVDSCommandParameters:{runAsync='true', hostId='5ae29ed2-9c0f-4754-8df3-b43d14726846', vmIds='[bcd5776e-978b-4723-933d-70a80322a796]'}), log id: 26b22afe 2017-10-01 16:37:28,739-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVDSCommand] (DefaultQuartzScheduler8) [4c8fa77d] FINISH, FullListVDSCommand, return: [{acpiEnable=true, emulatedMachine=pc-i440fx-rhel7.3.0, pid=109776, guestDiskMapping={}, transparentHugePages=true, timeOffset=-14400, cpuType=SandyBridge, smp=2, pauseCode=NOERR, guestNumaNodes=[Ljava.lang.Object;@4b69fd78, smartcardEnable=false, hypervEnable=true, custom={device_2c3e15c9-1aba-4575-8d80-08f6b5a4e972=VmDevice:{id='VmDeviceId:{deviceId='2c3e15c9-1aba-4575-8d80-08f6b5a4e972', vmId='bcd5776e-978b-4723-933d-70a80322a796'}', device='ide', type='CONTROLLER', bootOrder='0', specParams='[]', address='{slot=0x01, bus=0x00, domain=0x0000, type=pci, function=0x1}', managed='false', plugged='true', readOnly='false', deviceAlias='ide', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_2c3e15c9-1aba-4575-8d80-08f6b5a4e972device_002231b3-f832-41a0-920f-ffbed0a1e4cbdevice_bd98dcd7-8357-48c7-9c30-d5d175a5404adevice_f25c837c-fdf1-47c5-88ce-53e75efa9f13device_fff4e79f-0f92-419c-b778-961e316b2d64device_211c4b7b-605c-481c-8931-89cd68cf9f42device_e80fb0df-9d7d-4d57-8031-b86198e57969=VmDevice:{id='VmDeviceId:{deviceId='e80fb0df-9d7d-4d57-8031-b86198e57969', vmId='bcd5776e-978b-4723-933d-70a80322a796'}', device='spicevmc', type='REDIRDEV', bootOrder='0', specParams='[]', address='{bus=0, type=usb, port=3}', managed='false', plugged='true', readOnly='false', deviceAlias='redir2', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_2c3e15c9-1aba-4575-8d80-08f6b5a4e972device_002231b3-f832-41a0-920f-ffbed0a1e4cbdevice_bd98dcd7-8357-48c7-9c30-d5d175a5404adevice_f25c837c-fdf1-47c5-88ce-53e75efa9f13device_fff4e79f-0f92-419c-b778-961e316b2d64device_211c4b7b-605c-481c-8931-89cd68cf9f42=VmDevice:{id='VmDeviceId:{deviceId='211c4b7b-605c-481c-8931-89cd68cf9f42', vmId='bcd5776e-978b-4723-933d-70a80322a796'}', device='spicevmc', type='REDIRDEV', bootOrder='0', specParams='[]', address='{bus=0, type=usb, port=2}', managed='false', plugged='true', readOnly='false', deviceAlias='redir1', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_2c3e15c9-1aba-4575-8d80-08f6b5a4e972device_002231b3-f832-41a0-920f-ffbed0a1e4cbdevice_bd98dcd7-8357-48c7-9c30-d5d175a5404adevice_f25c837c-fdf1-47c5-88ce-53e75efa9f13=VmDevice:{id='VmDeviceId:{deviceId='f25c837c-fdf1-47c5-88ce-53e75efa9f13', vmId='bcd5776e-978b-4723-933d-70a80322a796'}', device='spicevmc', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=3}', managed='false', plugged='true', readOnly='false', deviceAlias='channel2', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_2c3e15c9-1aba-4575-8d80-08f6b5a4e972device_002231b3-f832-41a0-920f-ffbed0a1e4cb=VmDevice:{id='VmDeviceId:{deviceId='002231b3-f832-41a0-920f-ffbed0a1e4cb', vmId='bcd5776e-978b-4723-933d-70a80322a796'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=1}', managed='false', plugged='true', readOnly='false', deviceAlias='channel0', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_2c3e15c9-1aba-4575-8d80-08f6b5a4e972device_002231b3-f832-41a0-920f-ffbed0a1e4cbdevice_bd98dcd7-8357-48c7-9c30-d5d175a5404adevice_f25c837c-fdf1-47c5-88ce-53e75efa9f13device_fff4e79f-0f92-419c-b778-961e316b2d64device_211c4b7b-605c-481c-8931-89cd68cf9f42device_e80fb0df-9d7d-4d57-8031-b86198e57969device_a44c133d-b548-4dd3-be67-0fcbc1d244c4=VmDevice:{id='VmDeviceId:{deviceId='a44c133d-b548-4dd3-be67-0fcbc1d244c4', vmId='bcd5776e-978b-4723-933d-70a80322a796'}', device='spicevmc', type='REDIRDEV', bootOrder='0', specParams='[]', address='{bus=0, type=usb, port=4}', managed='false', plugged='true', readOnly='false', deviceAlias='redir3', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_2c3e15c9-1aba-4575-8d80-08f6b5a4e972device_002231b3-f832-41a0-920f-ffbed0a1e4cbdevice_bd98dcd7-8357-48c7-9c30-d5d175a5404a=VmDevice:{id='VmDeviceId:{deviceId='bd98dcd7-8357-48c7-9c30-d5d175a5404a', vmId='bcd5776e-978b-4723-933d-70a80322a796'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=2}', managed='false', plugged='true', readOnly='false', deviceAlias='channel1', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_2c3e15c9-1aba-4575-8d80-08f6b5a4e972device_002231b3-f832-41a0-920f-ffbed0a1e4cbdevice_bd98dcd7-8357-48c7-9c30-d5d175a5404adevice_f25c837c-fdf1-47c5-88ce-53e75efa9f13device_fff4e79f-0f92-419c-b778-961e316b2d64=VmDevice:{id='VmDeviceId:{deviceId='fff4e79f-0f92-419c-b778-961e316b2d64', vmId='bcd5776e-978b-4723-933d-70a80322a796'}', device='spicevmc', type='REDIRDEV', bootOrder='0', specParams='[]', address='{bus=0, type=usb, port=1}', managed='false', plugged='true', readOnly='false', deviceAlias='redir0', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}}, vmType=kvm, memSize=8192, smpCoresPerSocket=1, vmName=cape, nice=0, status=Up, bootMenuEnable=false, vmId=bcd5776e-978b-4723-933d-70a80322a796, smpThreadsPerCore=1, memGuaranteedSize=5461, kvmEnable=true, displayNetwork=yorku, devices=[Ljava.lang.Object;@1f36994d, display=qxl, maxVCpus=16, clientIp=, statusTime=5257553400, maxMemSlots=16}], log id: 26b22afe 2017-10-01 16:37:28,739-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler8) [4c8fa77d] Received a spicevmc Device without an address when processing VM bcd5776e-978b-4723-933d-70a80322a796 devices, skipping device: {device=spicevmc, bus=usb, type=redir, deviceId=8241e2e6-7340-40c8-a4f1-92e4ccff440c, specParams={}} 2017-10-01 16:37:28,740-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler8) [4c8fa77d] Received a spicevmc Device without an address when processing VM bcd5776e-978b-4723-933d-70a80322a796 devices, skipping device: {device=spicevmc, bus=usb, type=redir, deviceId=9a03ca98-987d-4309-bd8b-20fd230c9057, specParams={}} 2017-10-01 16:37:28,740-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler8) [4c8fa77d] Received a spicevmc Device without an address when processing VM bcd5776e-978b-4723-933d-70a80322a796 devices, skipping device: {device=spicevmc, bus=usb, type=redir, deviceId=f60084fb-a381-49f3-8006-d45bc17e0312, specParams={}} 2017-10-01 16:37:28,740-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler8) [4c8fa77d] Received a spicevmc Device without an address when processing VM bcd5776e-978b-4723-933d-70a80322a796 devices, skipping device: {device=spicevmc, bus=usb, type=redir, deviceId=48522fd4-dfea-48f7-83a4-9c574b884f77, specParams={}} 2017-10-01 16:37:28,740-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler8) [4c8fa77d] Received a spice Device without an address when processing VM bcd5776e-978b-4723-933d-70a80322a796 devices, skipping device: {device=spice, specParams={fileTransferEnable=true, displayNetwork=yorku, copyPasteEnable=true, displayIp=130.63.94.37, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,ssmartcard,susbredir}, type=graphics, deviceId=a09ffaa5-fede-473c-a53a-986424dce8e4, tlsPort=5910} 2017-10-01 16:37:28,740-04 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler8) [4c8fa77d] VM 'bcd5776e-978b-4723-933d-70a80322a796' managed non pluggable device was removed unexpectedly from libvirt: 'VmDevice:{id='VmDeviceId:{deviceId='48522fd4-dfea-48f7-83a4-9c574b884f77', vmId='bcd5776e-978b-4723-933d-70a80322a796'}', device='spicevmc', type='REDIR', bootOrder='0', specParams='[]', address='', managed='true', plugged='false', readOnly='false', deviceAlias='', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}' 2017-10-01 16:37:28,740-04 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler8) [4c8fa77d] VM 'bcd5776e-978b-4723-933d-70a80322a796' managed non pluggable device was removed unexpectedly from libvirt: 'VmDevice:{id='VmDeviceId:{deviceId='8241e2e6-7340-40c8-a4f1-92e4ccff440c', vmId='bcd5776e-978b-4723-933d-70a80322a796'}', device='spicevmc', type='REDIR', bootOrder='0', specParams='[]', address='', managed='true', plugged='false', readOnly='false', deviceAlias='', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}' 2017-10-01 16:37:28,740-04 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler8) [4c8fa77d] VM 'bcd5776e-978b-4723-933d-70a80322a796' managed non pluggable device was removed unexpectedly from libvirt: 'VmDevice:{id='VmDeviceId:{deviceId='9a03ca98-987d-4309-bd8b-20fd230c9057', vmId='bcd5776e-978b-4723-933d-70a80322a796'}', device='spicevmc', type='REDIR', bootOrder='0', specParams='[]', address='', managed='true', plugged='false', readOnly='false', deviceAlias='', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}' 2017-10-01 16:37:28,740-04 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler8) [4c8fa77d] VM 'bcd5776e-978b-4723-933d-70a80322a796' managed non pluggable device was removed unexpectedly from libvirt: 'VmDevice:{id='VmDeviceId:{deviceId='f60084fb-a381-49f3-8006-d45bc17e0312', vmId='bcd5776e-978b-4723-933d-70a80322a796'}', device='spicevmc', type='REDIR', bootOrder='0', specParams='[]', address='', managed='true', plugged='false', readOnly='false', deviceAlias='', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}' 2017-10-01 16:37:31,357-04 INFO [org.ovirt.engine.core.bll.RunVmCommand] (default task-75) [2aed97dc-0bbd-4cd4-b75e-5aa411331429] Lock Acquired to object 'EngineLock:{exclusiveLocks='[ddf37076-e795-4c87-a4fb-9efcaae76d72=VM]', sharedLocks=''}' 2017-10-01 16:37:31,373-04 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (default task-75) [2aed97dc-0bbd-4cd4-b75e-5aa411331429] START, IsVmDuringInitiatingVDSCommand( IsVmDuringInitiatingVDSCommandParameters:{runAsync='true', vmId='ddf37076-e795-4c87-a4fb-9efcaae76d72'}), log id: 6dadc413 2017-10-01 16:37:31,373-04 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (default task-75) [2aed97dc-0bbd-4cd4-b75e-5aa411331429] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 6dadc413 2017-10-01 16:37:31,455-04 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-6-thread-13) [2aed97dc-0bbd-4cd4-b75e-5aa411331429] Running command: RunVmCommand internal: false. Entities affected : ID: ddf37076-e795-4c87-a4fb-9efcaae76d72 Type: VMAction group RUN_VM with role type USER 2017-10-01 16:37:31,538-04 INFO [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (org.ovirt.thread.pool-6-thread-13) [2aed97dc-0bbd-4cd4-b75e-5aa411331429] START, UpdateVmDynamicDataVDSCommand( UpdateVmDynamicDataVDSCommandParameters:{runAsync='true', hostId='null', vmId='ddf37076-e795-4c87-a4fb-9efcaae76d72', vmDynamic='org.ovirt.engine.core.common.businessentities.VmDynamic@f50dd514'}), log id: 3621f89d 2017-10-01 16:37:31,540-04 INFO [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (org.ovirt.thread.pool-6-thread-13) [2aed97dc-0bbd-4cd4-b75e-5aa411331429] FINISH, UpdateVmDynamicDataVDSCommand, log id: 3621f89d 2017-10-01 16:37:31,542-04 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-6-thread-13) [2aed97dc-0bbd-4cd4-b75e-5aa411331429] START, CreateVmVDSCommand( CreateVmVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vmId='ddf37076-e795-4c87-a4fb-9efcaae76d72', vm='VM [dc1]'}), log id: 1632ea2 2017-10-01 16:37:31,543-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-6-thread-13) [2aed97dc-0bbd-4cd4-b75e-5aa411331429] START, CreateVDSCommand(HostName = virt3, CreateVmVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vmId='ddf37076-e795-4c87-a4fb-9efcaae76d72', vm='VM [dc1]'}), log id: 1b628c8 2017-10-01 16:37:31,547-04 INFO [org.ovirt.engine.core.vdsbroker.builder.vminfo.VmInfoBuilderImpl] (org.ovirt.thread.pool-6-thread-13) [2aed97dc-0bbd-4cd4-b75e-5aa411331429] Bootable disk '9cb78b91-fa60-4992-b586-c9537de7eacd' set to index '0' 2017-10-01 16:37:31,561-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-6-thread-13) [2aed97dc-0bbd-4cd4-b75e-5aa411331429] VM {memGuaranteedSize=2730, smpThreadsPerCore=1, cpuType=SandyBridge, vmId=ddf37076-e795-4c87-a4fb-9efcaae76d72, acpiEnable=true, vmType=kvm, smp=2, smpCoresPerSocket=1, emulatedMachine=pc-i440fx-rhel7.3.0, smartcardEnable=false, guestNumaNodes=[{memory=4096, cpus=0,1, nodeIndex=0}], transparentHugePages=true, displayNetwork=yorku, vmName=dc1, maxVCpus=16, kvmEnable=true, devices=[{address={bus=0x00, domain=0x0000, function=0x0, slot=0x02, type=pci}, type=video, specParams={vgamem=16384, heads=1, vram=8192, ram=65536}, device=qxl, deviceId=7b1004d9-dbbc-4a42-abc8-7b3eaf0d7da7}, {type=graphics, specParams={fileTransferEnable=true, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,ssmartcard,susbredir, copyPasteEnable=true}, device=spice, deviceId=6fd350b3-e6f6-46ec-981d-d0250aebb024}, {iface=ide, shared=false, path=, address={bus=1, controller=0, unit=0, type=drive, target=0}, readonly=true, index=2, type=disk, specParams={path=}, device=cdrom, deviceId=a684233f-b767-40fd-af0a-5ca18b7a267f}, {discard=false, shared=false, address={bus=0, controller=0, unit=0, type=drive, target=0}, imageID=9cb78b91-fa60-4992-b586-c9537de7eacd, format=raw, index=0, optional=false, type=disk, deviceId=9cb78b91-fa60-4992-b586-c9537de7eacd, domainID=c8653dba-63ec-4447-872c-0ffcc6af9c0f, propagateErrors=off, iface=scsi, readonly=false, bootOrder=1, poolID=aca61eac-ed8a-484b-bbee-3645148f6405, volumeID=c0e91d53-0e42-4ae6-ad68-2f4f78b5f2fb, specParams={}, device=disk}, {discard=false, shared=false, address={bus=0, controller=0, unit=1, type=drive, target=0}, imageID=62324a1d-adea-4d01-a9bb-14094b583c57, format=raw, optional=false, type=disk, deviceId=62324a1d-adea-4d01-a9bb-14094b583c57, domainID=c8653dba-63ec-4447-872c-0ffcc6af9c0f, propagateErrors=off, iface=scsi, readonly=false, poolID=aca61eac-ed8a-484b-bbee-3645148f6405, volumeID=060a88a9-9648-46af-ac9f-0b1a250f924e, specParams={}, device=disk}, {filter=vdsm-no-mac-spoofing, nicModel=pv, address={bus=0x00, domain=0x0000, function=0x0, slot=0x03, type=pci}, type=interface, specParams={inbound={}, outbound={}}, device=bridge, linkActive=true, deviceId=11d0bc3a-0d4c-4019-b3e2-f1feafd3f1a3, macAddr=00:1a:4a:32:3d:3c, network=yorku}, {filter=vdsm-no-mac-spoofing, nicModel=pv, address={bus=0x00, domain=0x0000, function=0x0, slot=0x04, type=pci}, type=interface, specParams={inbound={}, outbound={}}, device=bridge, linkActive=true, deviceId=3603df3e-d509-4751-9e46-ac7c02ebe6e8, macAddr=00:1a:4a:32:3d:3d, network=mrpriv}, {address={bus=0x00, domain=0x0000, function=0x2, slot=0x01, type=pci}, index=0, model=piix3-uhci, type=controller, specParams={}, device=usb, deviceId=131332a1-920a-4f6d-bf32-4ed81abc7e6e}, {address={bus=0x00, domain=0x0000, function=0x0, slot=0x05, type=pci}, index=0, model=virtio-scsi, type=controller, specParams={}, device=scsi, deviceId=01fb605f-82c7-4667-bbf5-0bec77e70d56}, {address={bus=0x00, domain=0x0000, function=0x0, slot=0x06, type=pci}, type=controller, specParams={}, device=virtio-serial, deviceId=496a204a-4852-454a-a15c-0cd67d06aa4c}, {model=virtio, type=rng, specParams={source=urandom}, device=virtio, deviceId=1cdae0ee-a6d3-44a4-9a4b-821ce6b16e14}], custom={device_24808fec-33b5-4e67-a36b-27a714902b1fdevice_04fce4fe-c529-44e2-ae96-a40cc6b9c248device_7fb8cc08-32d2-4614-b0aa-99a506eb243edevice_30a82b72-d749-4b0e-8a66-938549be5163=VmDevice:{id='VmDeviceId:{deviceId='30a82b72-d749-4b0e-8a66-938549be5163', vmId='ddf37076-e795-4c87-a4fb-9efcaae76d72'}', device='ide', type='CONTROLLER', bootOrder='0', specParams='[]', address='{slot=0x01, bus=0x00, domain=0x0000, type=pci, function=0x1}', managed='false', plugged='true', readOnly='false', deviceAlias='ide', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_24808fec-33b5-4e67-a36b-27a714902b1fdevice_04fce4fe-c529-44e2-ae96-a40cc6b9c248device_7fb8cc08-32d2-4614-b0aa-99a506eb243e=VmDevice:{id='VmDeviceId:{deviceId='7fb8cc08-32d2-4614-b0aa-99a506eb243e', vmId='ddf37076-e795-4c87-a4fb-9efcaae76d72'}', device='spicevmc', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=3}', managed='false', plugged='true', readOnly='false', deviceAlias='channel2', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_24808fec-33b5-4e67-a36b-27a714902b1fdevice_04fce4fe-c529-44e2-ae96-a40cc6b9c248=VmDevice:{id='VmDeviceId:{deviceId='04fce4fe-c529-44e2-ae96-a40cc6b9c248', vmId='ddf37076-e795-4c87-a4fb-9efcaae76d72'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=2}', managed='false', plugged='true', readOnly='false', deviceAlias='channel1', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_24808fec-33b5-4e67-a36b-27a714902b1f=VmDevice:{id='VmDeviceId:{deviceId='24808fec-33b5-4e67-a36b-27a714902b1f', vmId='ddf37076-e795-4c87-a4fb-9efcaae76d72'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=1}', managed='false', plugged='true', readOnly='false', deviceAlias='channel0', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}}, display=qxl, timeOffset=0, nice=0, maxMemSize=16384, maxMemSlots=16, bootMenuEnable=false, memSize=4096} 2017-10-01 16:37:31,576-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-6-thread-13) [2aed97dc-0bbd-4cd4-b75e-5aa411331429] FINISH, CreateVDSCommand, log id: 1b628c8 2017-10-01 16:37:31,590-04 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-6-thread-13) [2aed97dc-0bbd-4cd4-b75e-5aa411331429] FINISH, CreateVmVDSCommand, return: WaitForLaunch, log id: 1632ea2 2017-10-01 16:37:31,592-04 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-6-thread-13) [2aed97dc-0bbd-4cd4-b75e-5aa411331429] Lock freed to object 'EngineLock:{exclusiveLocks='[ddf37076-e795-4c87-a4fb-9efcaae76d72=VM]', sharedLocks=''}' 2017-10-01 16:37:31,600-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-13) [2aed97dc-0bbd-4cd4-b75e-5aa411331429] EVENT_ID: USER_STARTED_VM(153), Correlation ID: 2aed97dc-0bbd-4cd4-b75e-5aa411331429, Job ID: 912a1ff2-473d-482d-87f9-badae0f73591, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM dc1 was started by jas@eecs.yorku.ca-authz (Host: virt3). 2017-10-01 16:37:39,954-04 INFO [org.ovirt.engine.core.bll.RunVmCommand] (default task-85) [1344ab54-8b0b-438b-b372-f31b0497a034] Lock Acquired to object 'EngineLock:{exclusiveLocks='[4ba80442-b361-4026-ae8c-895fd28042b0=VM]', sharedLocks=''}' 2017-10-01 16:37:39,979-04 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (default task-85) [1344ab54-8b0b-438b-b372-f31b0497a034] START, IsVmDuringInitiatingVDSCommand( IsVmDuringInitiatingVDSCommandParameters:{runAsync='true', vmId='4ba80442-b361-4026-ae8c-895fd28042b0'}), log id: 6ceb2bb2 2017-10-01 16:37:39,980-04 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (default task-85) [1344ab54-8b0b-438b-b372-f31b0497a034] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 6ceb2bb2 2017-10-01 16:37:40,076-04 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-6-thread-23) [1344ab54-8b0b-438b-b372-f31b0497a034] Running command: RunVmCommand internal: false. Entities affected : ID: 4ba80442-b361-4026-ae8c-895fd28042b0 Type: VMAction group RUN_VM with role type USER 2017-10-01 16:37:40,160-04 INFO [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (org.ovirt.thread.pool-6-thread-23) [1344ab54-8b0b-438b-b372-f31b0497a034] START, UpdateVmDynamicDataVDSCommand( UpdateVmDynamicDataVDSCommandParameters:{runAsync='true', hostId='null', vmId='4ba80442-b361-4026-ae8c-895fd28042b0', vmDynamic='org.ovirt.engine.core.common.businessentities.VmDynamic@afd1a27e'}), log id: 2ec41702 2017-10-01 16:37:40,163-04 INFO [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (org.ovirt.thread.pool-6-thread-23) [1344ab54-8b0b-438b-b372-f31b0497a034] FINISH, UpdateVmDynamicDataVDSCommand, log id: 2ec41702 2017-10-01 16:37:40,165-04 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-6-thread-23) [1344ab54-8b0b-438b-b372-f31b0497a034] START, CreateVmVDSCommand( CreateVmVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vmId='4ba80442-b361-4026-ae8c-895fd28042b0', vm='VM [dist]'}), log id: 763dc7f4 2017-10-01 16:37:40,168-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-6-thread-23) [1344ab54-8b0b-438b-b372-f31b0497a034] START, CreateVDSCommand(HostName = virt2, CreateVmVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vmId='4ba80442-b361-4026-ae8c-895fd28042b0', vm='VM [dist]'}), log id: 62cd4211 2017-10-01 16:37:40,172-04 INFO [org.ovirt.engine.core.vdsbroker.builder.vminfo.VmInfoBuilderImpl] (org.ovirt.thread.pool-6-thread-23) [1344ab54-8b0b-438b-b372-f31b0497a034] Bootable disk '6a1e570c-e083-431e-ac09-ce769f5b5a59' set to index '0' 2017-10-01 16:37:40,186-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-6-thread-23) [1344ab54-8b0b-438b-b372-f31b0497a034] VM {memGuaranteedSize=682, smpThreadsPerCore=1, cpuType=SandyBridge, vmId=4ba80442-b361-4026-ae8c-895fd28042b0, acpiEnable=true, vmType=kvm, smp=1, smpCoresPerSocket=1, emulatedMachine=pc-i440fx-rhel7.3.0, smartcardEnable=false, guestNumaNodes=[{memory=1024, cpus=0, nodeIndex=0}], transparentHugePages=true, displayNetwork=yorku, vmName=dist, maxVCpus=16, kvmEnable=true, devices=[{address={bus=0x00, domain=0x0000, function=0x0, slot=0x02, type=pci}, type=video, specParams={heads=1, vram=32768}, device=qxl, deviceId=6cd39071-4c67-4bae-ac21-528cd7dffebe}, {type=graphics, specParams={fileTransferEnable=true, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,ssmartcard,susbredir, copyPasteEnable=true}, device=spice, deviceId=89976fb0-750d-404d-a52b-9c941a96559b}, {iface=ide, shared=false, path=, address={bus=1, controller=0, unit=0, type=drive, target=0}, readonly=true, index=2, type=disk, specParams={path=}, device=cdrom, deviceId=ab1fc5d5-ffc7-48bf-9db3-923206a7e8fd}, {discard=false, shared=false, address={bus=0x00, domain=0x0000, function=0x0, slot=0x07, type=pci}, imageID=6a1e570c-e083-431e-ac09-ce769f5b5a59, format=raw, index=0, optional=false, type=disk, deviceId=6a1e570c-e083-431e-ac09-ce769f5b5a59, domainID=c8653dba-63ec-4447-872c-0ffcc6af9c0f, propagateErrors=off, iface=virtio, readonly=false, bootOrder=1, poolID=aca61eac-ed8a-484b-bbee-3645148f6405, volumeID=02d9c2ae-9c86-41c1-a2bf-97595a50ddc1, specParams={}, device=disk}, {filter=vdsm-no-mac-spoofing, nicModel=pv, address={bus=0x00, domain=0x0000, function=0x0, slot=0x03, type=pci}, type=interface, specParams={inbound={}, outbound={}}, device=bridge, linkActive=true, deviceId=cb98f335-375b-4e10-98f9-38f2c7c771a5, macAddr=00:1a:4a:32:3d:27, network=yorku}, {filter=vdsm-no-mac-spoofing, nicModel=pv, address={bus=0x00, domain=0x0000, function=0x0, slot=0x04, type=pci}, type=interface, specParams={inbound={}, outbound={}}, device=bridge, linkActive=true, deviceId=d7f4d9f1-1a99-4e90-abc6-bfc6c53c5050, macAddr=00:1a:4a:32:3d:28, network=mrpriv}, {address={bus=0x00, domain=0x0000, function=0x2, slot=0x01, type=pci}, index=0, model=piix3-uhci, type=controller, specParams={}, device=usb, deviceId=120a0eb5-12f7-4418-b26d-2d9efe4fd522}, {address={bus=0x00, domain=0x0000, function=0x0, slot=0x05, type=pci}, index=0, model=virtio-scsi, type=controller, specParams={}, device=scsi, deviceId=09942c60-2b37-4d43-9576-a350c37ad722}, {address={bus=0x00, domain=0x0000, function=0x0, slot=0x06, type=pci}, type=controller, specParams={}, device=virtio-serial, deviceId=325f0bd7-8e98-462e-a552-c5ebf9296d1b}, {model=virtio, type=rng, specParams={source=urandom}, device=virtio, deviceId=74bb1521-3859-422b-8f2e-e71dab8e1352}], custom={device_72d1cb01-6d15-4697-8e1a-2eac5aab0384device_ad918a16-7cf9-44e9-a485-8444729780bfdevice_12e6af94-7afb-4038-851d-5bf4a1228a44device_b18c085f-f72c-47a0-96b7-17cc800660ef=VmDevice:{id='VmDeviceId:{deviceId='b18c085f-f72c-47a0-96b7-17cc800660ef', vmId='4ba80442-b361-4026-ae8c-895fd28042b0'}', device='ide', type='CONTROLLER', bootOrder='0', specParams='[]', address='{slot=0x01, bus=0x00, domain=0x0000, type=pci, function=0x1}', managed='false', plugged='true', readOnly='false', deviceAlias='ide', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_72d1cb01-6d15-4697-8e1a-2eac5aab0384=VmDevice:{id='VmDeviceId:{deviceId='72d1cb01-6d15-4697-8e1a-2eac5aab0384', vmId='4ba80442-b361-4026-ae8c-895fd28042b0'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=1}', managed='false', plugged='true', readOnly='false', deviceAlias='channel0', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_72d1cb01-6d15-4697-8e1a-2eac5aab0384device_ad918a16-7cf9-44e9-a485-8444729780bfdevice_12e6af94-7afb-4038-851d-5bf4a1228a44=VmDevice:{id='VmDeviceId:{deviceId='12e6af94-7afb-4038-851d-5bf4a1228a44', vmId='4ba80442-b361-4026-ae8c-895fd28042b0'}', device='spicevmc', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=3}', managed='false', plugged='true', readOnly='false', deviceAlias='channel2', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_72d1cb01-6d15-4697-8e1a-2eac5aab0384device_ad918a16-7cf9-44e9-a485-8444729780bf=VmDevice:{id='VmDeviceId:{deviceId='ad918a16-7cf9-44e9-a485-8444729780bf', vmId='4ba80442-b361-4026-ae8c-895fd28042b0'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=2}', managed='false', plugged='true', readOnly='false', deviceAlias='channel1', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}}, display=qxl, timeOffset=0, nice=0, maxMemSize=4096, maxMemSlots=16, bootMenuEnable=false, memSize=1024} 2017-10-01 16:37:41,140-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-6-thread-23) [1344ab54-8b0b-438b-b372-f31b0497a034] FINISH, CreateVDSCommand, log id: 62cd4211 2017-10-01 16:37:41,143-04 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-6-thread-23) [1344ab54-8b0b-438b-b372-f31b0497a034] FINISH, CreateVmVDSCommand, return: WaitForLaunch, log id: 763dc7f4 2017-10-01 16:37:41,145-04 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-6-thread-23) [1344ab54-8b0b-438b-b372-f31b0497a034] Lock freed to object 'EngineLock:{exclusiveLocks='[4ba80442-b361-4026-ae8c-895fd28042b0=VM]', sharedLocks=''}' 2017-10-01 16:37:41,152-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-23) [1344ab54-8b0b-438b-b372-f31b0497a034] EVENT_ID: USER_STARTED_VM(153), Correlation ID: 1344ab54-8b0b-438b-b372-f31b0497a034, Job ID: 58794d11-7b5b-4217-83e7-091d0b61f128, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM dist was started by jas@eecs.yorku.ca-authz (Host: virt2). 2017-10-01 16:37:46,273-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmsStatisticsFetcher] (DefaultQuartzScheduler5) [137d41cf] Fetched 10 VMs from VDS 'e1e71e9d-a0cd-47db-96c6-f724f6cea736' 2017-10-01 16:37:46,274-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler5) [137d41cf] VM 'ddf37076-e795-4c87-a4fb-9efcaae76d72'(dc1) moved from 'WaitForLaunch' --> 'PoweringUp' 2017-10-01 16:37:46,286-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVDSCommand] (DefaultQuartzScheduler5) [137d41cf] START, FullListVDSCommand(HostName = virt3, FullListVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vmIds='[ddf37076-e795-4c87-a4fb-9efcaae76d72]'}), log id: 476e6a18 2017-10-01 16:37:47,300-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVDSCommand] (DefaultQuartzScheduler5) [137d41cf] FINISH, FullListVDSCommand, return: [{acpiEnable=true, emulatedMachine=pc-i440fx-rhel7.3.0, vmId=ddf37076-e795-4c87-a4fb-9efcaae76d72, guestDiskMapping={}, transparentHugePages=true, timeOffset=0, cpuType=SandyBridge, smp=2, pauseCode=NOERR, guestNumaNodes=[Ljava.lang.Object;@35fbbcbc, smartcardEnable=false, custom={device_24808fec-33b5-4e67-a36b-27a714902b1fdevice_04fce4fe-c529-44e2-ae96-a40cc6b9c248device_7fb8cc08-32d2-4614-b0aa-99a506eb243e=VmDevice:{id='VmDeviceId:{deviceId='7fb8cc08-32d2-4614-b0aa-99a506eb243e', vmId='ddf37076-e795-4c87-a4fb-9efcaae76d72'}', device='spicevmc', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=3}', managed='false', plugged='true', readOnly='false', deviceAlias='channel2', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_24808fec-33b5-4e67-a36b-27a714902b1fdevice_04fce4fe-c529-44e2-ae96-a40cc6b9c248device_7fb8cc08-32d2-4614-b0aa-99a506eb243edevice_30a82b72-d749-4b0e-8a66-938549be5163=VmDevice:{id='VmDeviceId:{deviceId='30a82b72-d749-4b0e-8a66-938549be5163', vmId='ddf37076-e795-4c87-a4fb-9efcaae76d72'}', device='ide', type='CONTROLLER', bootOrder='0', specParams='[]', address='{slot=0x01, bus=0x00, domain=0x0000, type=pci, function=0x1}', managed='false', plugged='true', readOnly='false', deviceAlias='ide', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_24808fec-33b5-4e67-a36b-27a714902b1fdevice_04fce4fe-c529-44e2-ae96-a40cc6b9c248=VmDevice:{id='VmDeviceId:{deviceId='04fce4fe-c529-44e2-ae96-a40cc6b9c248', vmId='ddf37076-e795-4c87-a4fb-9efcaae76d72'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=2}', managed='false', plugged='true', readOnly='false', deviceAlias='channel1', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_24808fec-33b5-4e67-a36b-27a714902b1f=VmDevice:{id='VmDeviceId:{deviceId='24808fec-33b5-4e67-a36b-27a714902b1f', vmId='ddf37076-e795-4c87-a4fb-9efcaae76d72'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=1}', managed='false', plugged='true', readOnly='false', deviceAlias='channel0', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}}, vmType=kvm, memSize=4096, smpCoresPerSocket=1, vmName=dc1, nice=0, status=Up, maxMemSize=16384, bootMenuEnable=false, pid=8928, smpThreadsPerCore=1, memGuaranteedSize=2730, kvmEnable=true, displayNetwork=yorku, devices=[Ljava.lang.Object;@58ad0da, display=qxl, maxVCpus=16, clientIp=, statusTime=5262882740, maxMemSlots=16}], log id: 476e6a18 2017-10-01 16:37:47,304-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler5) [137d41cf] Received a memballoon Device without an address when processing VM ddf37076-e795-4c87-a4fb-9efcaae76d72 devices, skipping device: {device=memballoon, specParams={model=none}, type=balloon, alias=balloon0} 2017-10-01 16:37:47,304-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler5) [137d41cf] Received a spice Device without an address when processing VM ddf37076-e795-4c87-a4fb-9efcaae76d72 devices, skipping device: {device=spice, specParams={fileTransferEnable=true, displayNetwork=yorku, copyPasteEnable=true, displayIp=130.63.94.36, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,ssmartcard,susbredir}, type=graphics, deviceId=6fd350b3-e6f6-46ec-981d-d0250aebb024, tlsPort=5909} 2017-10-01 16:37:56,206-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmsStatisticsFetcher] (DefaultQuartzScheduler5) [58395886] Fetched 6 VMs from VDS 'a0ccfd11-3402-4eba-a119-4987561b738e' 2017-10-01 16:37:56,208-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler5) [58395886] VM '4ba80442-b361-4026-ae8c-895fd28042b0'(dist) moved from 'WaitForLaunch' --> 'PoweringUp' 2017-10-01 16:37:56,215-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVDSCommand] (DefaultQuartzScheduler5) [58395886] START, FullListVDSCommand(HostName = virt2, FullListVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vmIds='[4ba80442-b361-4026-ae8c-895fd28042b0]'}), log id: 29dad8 2017-10-01 16:37:56,247-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVDSCommand] (DefaultQuartzScheduler5) [58395886] FINISH, FullListVDSCommand, return: [{acpiEnable=true, emulatedMachine=pc-i440fx-rhel7.3.0, vmId=4ba80442-b361-4026-ae8c-895fd28042b0, guestDiskMapping={}, transparentHugePages=true, timeOffset=0, cpuType=SandyBridge, smp=1, pauseCode=NOERR, guestNumaNodes=[Ljava.lang.Object;@2d2a14eb, smartcardEnable=false, custom={device_72d1cb01-6d15-4697-8e1a-2eac5aab0384device_ad918a16-7cf9-44e9-a485-8444729780bfdevice_12e6af94-7afb-4038-851d-5bf4a1228a44=VmDevice:{id='VmDeviceId:{deviceId='12e6af94-7afb-4038-851d-5bf4a1228a44', vmId='4ba80442-b361-4026-ae8c-895fd28042b0'}', device='spicevmc', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=3}', managed='false', plugged='true', readOnly='false', deviceAlias='channel2', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_72d1cb01-6d15-4697-8e1a-2eac5aab0384=VmDevice:{id='VmDeviceId:{deviceId='72d1cb01-6d15-4697-8e1a-2eac5aab0384', vmId='4ba80442-b361-4026-ae8c-895fd28042b0'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=1}', managed='false', plugged='true', readOnly='false', deviceAlias='channel0', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_72d1cb01-6d15-4697-8e1a-2eac5aab0384device_ad918a16-7cf9-44e9-a485-8444729780bfdevice_12e6af94-7afb-4038-851d-5bf4a1228a44device_b18c085f-f72c-47a0-96b7-17cc800660ef=VmDevice:{id='VmDeviceId:{deviceId='b18c085f-f72c-47a0-96b7-17cc800660ef', vmId='4ba80442-b361-4026-ae8c-895fd28042b0'}', device='ide', type='CONTROLLER', bootOrder='0', specParams='[]', address='{slot=0x01, bus=0x00, domain=0x0000, type=pci, function=0x1}', managed='false', plugged='true', readOnly='false', deviceAlias='ide', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_72d1cb01-6d15-4697-8e1a-2eac5aab0384device_ad918a16-7cf9-44e9-a485-8444729780bf=VmDevice:{id='VmDeviceId:{deviceId='ad918a16-7cf9-44e9-a485-8444729780bf', vmId='4ba80442-b361-4026-ae8c-895fd28042b0'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=2}', managed='false', plugged='true', readOnly='false', deviceAlias='channel1', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}}, vmType=kvm, memSize=1024, smpCoresPerSocket=1, vmName=dist, nice=0, status=Up, maxMemSize=4096, bootMenuEnable=false, pid=4594, smpThreadsPerCore=1, memGuaranteedSize=682, kvmEnable=true, displayNetwork=yorku, devices=[Ljava.lang.Object;@795a1b4e, display=qxl, maxVCpus=16, clientIp=, statusTime=4726959450, maxMemSlots=16}], log id: 29dad8 2017-10-01 16:37:56,248-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler5) [58395886] Received a memballoon Device without an address when processing VM 4ba80442-b361-4026-ae8c-895fd28042b0 devices, skipping device: {device=memballoon, specParams={model=none}, type=balloon, alias=balloon0} 2017-10-01 16:37:56,248-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler5) [58395886] Received a spice Device without an address when processing VM 4ba80442-b361-4026-ae8c-895fd28042b0 devices, skipping device: {device=spice, specParams={fileTransferEnable=true, displayNetwork=yorku, copyPasteEnable=true, displayIp=130.63.94.35, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,ssmartcard,susbredir}, type=graphics, deviceId=89976fb0-750d-404d-a52b-9c941a96559b, tlsPort=5905} 2017-10-01 16:37:57,027-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [4729f1e4] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 16:37:58,797-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler8) [57e05860] VM 'dc66c051-24b7-4cb8-99eb-db398d396ef1'(samba1) moved from 'PoweringUp' --> 'Up' 2017-10-01 16:37:58,800-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler8) [57e05860] VM '77766cb4-0625-4432-a16f-def5e702102a'(labtest1) moved from 'PoweringUp' --> 'Up' 2017-10-01 16:37:58,812-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [57e05860] EVENT_ID: USER_RUN_VM(32), Correlation ID: 61618ff4-e812-4539-8fce-28f9727ccce9, Job ID: 43c55b0d-d094-4a9f-9a56-6f56f06ea094, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM samba1 started on Host virt4 2017-10-01 16:37:58,817-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [57e05860] EVENT_ID: USER_RUN_VM(32), Correlation ID: 24749604-39d1-4a75-9d4c-2da6787b045b, Job ID: 8c7ee292-ecc8-4358-859f-ec331bf1a800, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM labtest1 started on Host virt4 2017-10-01 16:38:13,833-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler9) [2d8de09e] VM 'bcd5776e-978b-4723-933d-70a80322a796'(cape) moved from 'PoweringUp' --> 'Up' 2017-10-01 16:38:13,846-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler9) [2d8de09e] EVENT_ID: USER_RUN_VM(32), Correlation ID: c91b7cff-5a83-4833-822d-c3f0de418e5a, Job ID: d4f6261c-5ada-4f21-996e-263cb267c54e, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM cape started on Host virt4 2017-10-01 16:38:32,385-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler5) [6a851e0f] VM 'ddf37076-e795-4c87-a4fb-9efcaae76d72'(dc1) moved from 'PoweringUp' --> 'Up' 2017-10-01 16:38:32,406-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [6a851e0f] EVENT_ID: USER_RUN_VM(32), Correlation ID: 2aed97dc-0bbd-4cd4-b75e-5aa411331429, Job ID: 912a1ff2-473d-482d-87f9-badae0f73591, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM dc1 started on Host virt3 2017-10-01 16:38:43,485-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler10) [641e097d] VM '4ba80442-b361-4026-ae8c-895fd28042b0'(dist) moved from 'PoweringUp' --> 'Up' 2017-10-01 16:38:43,494-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [641e097d] EVENT_ID: USER_RUN_VM(32), Correlation ID: 1344ab54-8b0b-438b-b372-f31b0497a034, Job ID: 58794d11-7b5b-4217-83e7-091d0b61f128, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM dist started on Host virt2 2017-10-01 16:38:57,048-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [62c2f74e] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 16:39:57,067-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [39eed51c] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 16:40:57,088-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [2a40e638] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 16:41:57,108-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [3ba8aa9b] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 16:42:57,132-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [42d2b17d] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 16:43:57,151-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [310bf5ca] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 16:44:57,172-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [7965b69e] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 16:45:57,198-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [2f616acf] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 16:46:33,496-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [32c6246e] EVENT_ID: VM_CONSOLE_DISCONNECTED(168), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: User admin@internal got disconnected from VM archive. 2017-10-01 16:46:57,220-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [265c7c39] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 16:47:57,244-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler7) [23a9f9c7] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 16:48:57,264-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [3b217a2] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 16:49:57,281-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [62c26cf0] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 16:50:57,300-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [2bb83fb2] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 16:51:57,319-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [585c25c0] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 16:52:57,345-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [3dd90521] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 16:53:57,365-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [27ef6fe3] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 16:54:57,384-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [156da261] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 16:55:57,404-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [4c3e475d] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 16:56:57,435-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [44cb1df0] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 16:57:57,461-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [65640f57] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 16:58:57,484-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [4d5bb565] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 16:59:57,506-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [365a0044] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 17:00:57,526-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [65b5a7b] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 17:01:57,548-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler7) [48beffd9] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 17:01:57,564-04 INFO [org.ovirt.engine.core.sso.servlets.OAuthRevokeServlet] (default task-359) [] User admin@internal successfully logged out 2017-10-01 17:01:57,579-04 INFO [org.ovirt.engine.core.bll.aaa.TerminateSessionsForTokenCommand] (default task-352) [262f46c4] Running command: TerminateSessionsForTokenCommand internal: true. 2017-10-01 17:02:57,607-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [c92c57b] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 17:03:57,630-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [6d040dbf] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 17:04:57,649-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [2240b91d] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 17:05:57,668-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [789d7307] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 17:06:57,689-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [739aa033] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 17:07:57,712-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [39acde0c] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 17:08:57,729-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [1cc7a9d4] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 17:09:57,749-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [4c1791c3] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 17:10:57,772-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [ba5921f] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 17:11:57,792-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [4173a722] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 17:12:57,815-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [5d7c75e1] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 17:13:57,841-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [30a1e731] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 17:14:57,863-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [1786a8e2] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 17:15:57,883-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler7) [4bb3b795] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 17:16:57,903-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [608d20a9] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 17:17:57,929-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [4acc47c1] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 17:18:57,948-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [1d4071d1] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 17:19:57,972-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [57941e21] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 17:20:30,528-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.OvfDataUpdater] (DefaultQuartzScheduler9) [599c42cd] Attempting to update VMs/Templates Ovf. 2017-10-01 17:20:30,531-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler9) [b3e573c] Before acquiring and wait lock 'EngineLock:{exclusiveLocks='[aca61eac-ed8a-484b-bbee-3645148f6405=OVF_UPDATE]', sharedLocks=''}' 2017-10-01 17:20:30,531-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler9) [b3e573c] Lock-wait acquired to object 'EngineLock:{exclusiveLocks='[aca61eac-ed8a-484b-bbee-3645148f6405=OVF_UPDATE]', sharedLocks=''}' 2017-10-01 17:20:30,532-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler9) [b3e573c] Running command: ProcessOvfUpdateForStoragePoolCommand internal: true. Entities affected : ID: aca61eac-ed8a-484b-bbee-3645148f6405 Type: StoragePool 2017-10-01 17:20:30,538-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler9) [b3e573c] Attempting to update VM OVFs in Data Center 'EECS-Primary' 2017-10-01 17:20:31,059-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler9) [b3e573c] Successfully updated VM OVFs in Data Center 'EECS-Primary' 2017-10-01 17:20:31,059-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler9) [b3e573c] Attempting to update template OVFs in Data Center 'EECS-Primary' 2017-10-01 17:20:31,060-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler9) [b3e573c] Successfully updated templates OVFs in Data Center 'EECS-Primary' 2017-10-01 17:20:31,060-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler9) [b3e573c] Attempting to remove unneeded template/vm OVFs in Data Center 'EECS-Primary' 2017-10-01 17:20:31,062-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler9) [b3e573c] Successfully removed unneeded template/vm OVFs in Data Center 'EECS-Primary' 2017-10-01 17:20:31,064-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler9) [b3e573c] Lock freed to object 'EngineLock:{exclusiveLocks='[aca61eac-ed8a-484b-bbee-3645148f6405=OVF_UPDATE]', sharedLocks=''}' 2017-10-01 17:20:31,086-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStorageDomainCommand] (DefaultQuartzScheduler9) [3fab6aa9] Lock Acquired to object 'EngineLock:{exclusiveLocks='[c8653dba-63ec-4447-872c-0ffcc6af9c0f=STORAGE]', sharedLocks='[aca61eac-ed8a-484b-bbee-3645148f6405=OVF_UPDATE]'}' 2017-10-01 17:20:31,120-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStorageDomainCommand] (DefaultQuartzScheduler9) [3fab6aa9] Running command: ProcessOvfUpdateForStorageDomainCommand internal: true. Entities affected : ID: c8653dba-63ec-4447-872c-0ffcc6af9c0f Type: StorageAction group MANIPULATE_STORAGE_DOMAIN with role type ADMIN 2017-10-01 17:20:31,135-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler9) [3fab6aa9] START, SetVolumeDescriptionVDSCommand( SetVolumeDescriptionVDSCommandParameters:{runAsync='true', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', ignoreFailoverLimit='false', storageDomainId='c8653dba-63ec-4447-872c-0ffcc6af9c0f', imageGroupId='bbbcf706-d019-4cc5-8096-ddf6d9436553', imageId='6c73cd57-01c6-4541-8157-495e5182f916'}), log id: 1bdbfb51 2017-10-01 17:20:31,136-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler9) [3fab6aa9] -- executeIrsBrokerCommand: calling 'setVolumeDescription', parameters: 2017-10-01 17:20:31,136-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler9) [3fab6aa9] ++ spUUID=aca61eac-ed8a-484b-bbee-3645148f6405 2017-10-01 17:20:31,136-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler9) [3fab6aa9] ++ sdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f 2017-10-01 17:20:31,136-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler9) [3fab6aa9] ++ imageGroupGUID=bbbcf706-d019-4cc5-8096-ddf6d9436553 2017-10-01 17:20:31,136-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler9) [3fab6aa9] ++ volUUID=6c73cd57-01c6-4541-8157-495e5182f916 2017-10-01 17:20:31,136-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler9) [3fab6aa9] ++ description={"Updated":false,"Last Updated":"Sun Oct 01 15:20:24 EDT 2017","Storage Domains":[{"uuid":"c8653dba-63ec-4447-872c-0ffcc6af9c0f"}],"Disk Description":"OVF_STORE"} 2017-10-01 17:20:31,900-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler9) [3fab6aa9] FINISH, SetVolumeDescriptionVDSCommand, log id: 1bdbfb51 2017-10-01 17:20:31,909-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.UploadStreamCommand] (DefaultQuartzScheduler9) [3fab6aa9] Lock Acquired to object 'EngineLock:{exclusiveLocks='', sharedLocks='[45f8b331-842e-48e7-9df8-56adddb93836=VDS_EXECUTION]'}' 2017-10-01 17:20:31,925-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.UploadStreamCommand] (DefaultQuartzScheduler9) [3fab6aa9] Running command: UploadStreamCommand internal: true. Entities affected : ID: c8653dba-63ec-4447-872c-0ffcc6af9c0f Type: Storage 2017-10-01 17:20:31,925-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (DefaultQuartzScheduler9) [3fab6aa9] START, UploadStreamVDSCommand(HostName = virt1, UploadStreamVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836'}), log id: 73ae1105 2017-10-01 17:20:31,925-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (DefaultQuartzScheduler9) [3fab6aa9] -- executeVdsBrokerCommand, parameters: 2017-10-01 17:20:31,925-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (DefaultQuartzScheduler9) [3fab6aa9] ++ spUUID=aca61eac-ed8a-484b-bbee-3645148f6405 2017-10-01 17:20:31,925-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (DefaultQuartzScheduler9) [3fab6aa9] ++ sdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f 2017-10-01 17:20:31,925-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (DefaultQuartzScheduler9) [3fab6aa9] ++ imageGUID=bbbcf706-d019-4cc5-8096-ddf6d9436553 2017-10-01 17:20:31,925-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (DefaultQuartzScheduler9) [3fab6aa9] ++ volUUID=6c73cd57-01c6-4541-8157-495e5182f916 2017-10-01 17:20:31,925-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (DefaultQuartzScheduler9) [3fab6aa9] ++ size=481280 2017-10-01 17:20:33,176-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (DefaultQuartzScheduler9) [3fab6aa9] FINISH, UploadStreamVDSCommand, log id: 73ae1105 2017-10-01 17:20:33,190-04 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler9) [3fab6aa9] CommandAsyncTask::Adding CommandMultiAsyncTasks object for command '8b1db5d0-e304-4985-9d55-ee08708e2440' 2017-10-01 17:20:33,190-04 INFO [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (DefaultQuartzScheduler9) [3fab6aa9] CommandMultiAsyncTasks::attachTask: Attaching task '76f839c1-f1d7-487f-8221-d43539c8f103' to command '8b1db5d0-e304-4985-9d55-ee08708e2440'. 2017-10-01 17:20:33,203-04 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler9) [3fab6aa9] Adding task '76f839c1-f1d7-487f-8221-d43539c8f103' (Parent Command 'UploadStream', Parameters Type 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters'), polling hasn't started yet.. 2017-10-01 17:20:33,208-04 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler9) [3fab6aa9] BaseAsyncTask::startPollingTask: Starting to poll task '76f839c1-f1d7-487f-8221-d43539c8f103'. 2017-10-01 17:20:33,212-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.UploadStreamCommand] (DefaultQuartzScheduler9) [3fab6aa9] Lock freed to object 'EngineLock:{exclusiveLocks='', sharedLocks='[45f8b331-842e-48e7-9df8-56adddb93836=VDS_EXECUTION]'}' 2017-10-01 17:20:33,214-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler9) [3fab6aa9] START, SetVolumeDescriptionVDSCommand( SetVolumeDescriptionVDSCommandParameters:{runAsync='true', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', ignoreFailoverLimit='false', storageDomainId='c8653dba-63ec-4447-872c-0ffcc6af9c0f', imageGroupId='bbbcf706-d019-4cc5-8096-ddf6d9436553', imageId='6c73cd57-01c6-4541-8157-495e5182f916'}), log id: 4f21772a 2017-10-01 17:20:33,214-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler9) [3fab6aa9] -- executeIrsBrokerCommand: calling 'setVolumeDescription', parameters: 2017-10-01 17:20:33,214-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler9) [3fab6aa9] ++ spUUID=aca61eac-ed8a-484b-bbee-3645148f6405 2017-10-01 17:20:33,214-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler9) [3fab6aa9] ++ sdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f 2017-10-01 17:20:33,214-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler9) [3fab6aa9] ++ imageGroupGUID=bbbcf706-d019-4cc5-8096-ddf6d9436553 2017-10-01 17:20:33,214-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler9) [3fab6aa9] ++ volUUID=6c73cd57-01c6-4541-8157-495e5182f916 2017-10-01 17:20:33,214-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler9) [3fab6aa9] ++ description={"Updated":true,"Size":481280,"Last Updated":"Sun Oct 01 17:20:31 EDT 2017","Storage Domains":[{"uuid":"c8653dba-63ec-4447-872c-0ffcc6af9c0f"}],"Disk Description":"OVF_STORE"} 2017-10-01 17:20:33,680-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler9) [3fab6aa9] FINISH, SetVolumeDescriptionVDSCommand, log id: 4f21772a 2017-10-01 17:20:33,682-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler9) [3fab6aa9] START, SetVolumeDescriptionVDSCommand( SetVolumeDescriptionVDSCommandParameters:{runAsync='true', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', ignoreFailoverLimit='false', storageDomainId='c8653dba-63ec-4447-872c-0ffcc6af9c0f', imageGroupId='def9e7db-b58b-4df1-ad2b-c47dc60c4303', imageId='357577c1-8d1b-4fad-b470-8e55341e26f6'}), log id: 787b353c 2017-10-01 17:20:33,682-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler9) [3fab6aa9] -- executeIrsBrokerCommand: calling 'setVolumeDescription', parameters: 2017-10-01 17:20:33,682-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler9) [3fab6aa9] ++ spUUID=aca61eac-ed8a-484b-bbee-3645148f6405 2017-10-01 17:20:33,682-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler9) [3fab6aa9] ++ sdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f 2017-10-01 17:20:33,682-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler9) [3fab6aa9] ++ imageGroupGUID=def9e7db-b58b-4df1-ad2b-c47dc60c4303 2017-10-01 17:20:33,682-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler9) [3fab6aa9] ++ volUUID=357577c1-8d1b-4fad-b470-8e55341e26f6 2017-10-01 17:20:33,682-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler9) [3fab6aa9] ++ description={"Updated":false,"Last Updated":"Sun Oct 01 15:20:24 EDT 2017","Storage Domains":[{"uuid":"c8653dba-63ec-4447-872c-0ffcc6af9c0f"}],"Disk Description":"OVF_STORE"} 2017-10-01 17:20:34,731-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler9) [3fab6aa9] FINISH, SetVolumeDescriptionVDSCommand, log id: 787b353c 2017-10-01 17:20:34,740-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.UploadStreamCommand] (DefaultQuartzScheduler9) [3fab6aa9] Lock Acquired to object 'EngineLock:{exclusiveLocks='', sharedLocks='[45f8b331-842e-48e7-9df8-56adddb93836=VDS_EXECUTION]'}' 2017-10-01 17:20:34,758-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.UploadStreamCommand] (DefaultQuartzScheduler9) [3fab6aa9] Running command: UploadStreamCommand internal: true. Entities affected : ID: c8653dba-63ec-4447-872c-0ffcc6af9c0f Type: Storage 2017-10-01 17:20:34,759-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (DefaultQuartzScheduler9) [3fab6aa9] START, UploadStreamVDSCommand(HostName = virt1, UploadStreamVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836'}), log id: 54dfa66a 2017-10-01 17:20:34,759-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (DefaultQuartzScheduler9) [3fab6aa9] -- executeVdsBrokerCommand, parameters: 2017-10-01 17:20:34,759-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (DefaultQuartzScheduler9) [3fab6aa9] ++ spUUID=aca61eac-ed8a-484b-bbee-3645148f6405 2017-10-01 17:20:34,759-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (DefaultQuartzScheduler9) [3fab6aa9] ++ sdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f 2017-10-01 17:20:34,759-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (DefaultQuartzScheduler9) [3fab6aa9] ++ imageGUID=def9e7db-b58b-4df1-ad2b-c47dc60c4303 2017-10-01 17:20:34,759-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (DefaultQuartzScheduler9) [3fab6aa9] ++ volUUID=357577c1-8d1b-4fad-b470-8e55341e26f6 2017-10-01 17:20:34,759-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (DefaultQuartzScheduler9) [3fab6aa9] ++ size=481280 2017-10-01 17:20:35,986-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (DefaultQuartzScheduler9) [3fab6aa9] FINISH, UploadStreamVDSCommand, log id: 54dfa66a 2017-10-01 17:20:36,000-04 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler9) [3fab6aa9] CommandAsyncTask::Adding CommandMultiAsyncTasks object for command '21fda0bd-eb44-4a0f-b81d-2e0fc1c492bc' 2017-10-01 17:20:36,000-04 INFO [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (DefaultQuartzScheduler9) [3fab6aa9] CommandMultiAsyncTasks::attachTask: Attaching task '2e1e9f29-e4f8-4e2c-86bb-406e9e095bd4' to command '21fda0bd-eb44-4a0f-b81d-2e0fc1c492bc'. 2017-10-01 17:20:36,016-04 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler9) [3fab6aa9] Adding task '2e1e9f29-e4f8-4e2c-86bb-406e9e095bd4' (Parent Command 'UploadStream', Parameters Type 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters'), polling hasn't started yet.. 2017-10-01 17:20:36,021-04 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler9) [3fab6aa9] BaseAsyncTask::startPollingTask: Starting to poll task '2e1e9f29-e4f8-4e2c-86bb-406e9e095bd4'. 2017-10-01 17:20:36,025-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.UploadStreamCommand] (DefaultQuartzScheduler9) [3fab6aa9] Lock freed to object 'EngineLock:{exclusiveLocks='', sharedLocks='[45f8b331-842e-48e7-9df8-56adddb93836=VDS_EXECUTION]'}' 2017-10-01 17:20:36,027-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler9) [3fab6aa9] START, SetVolumeDescriptionVDSCommand( SetVolumeDescriptionVDSCommandParameters:{runAsync='true', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', ignoreFailoverLimit='false', storageDomainId='c8653dba-63ec-4447-872c-0ffcc6af9c0f', imageGroupId='def9e7db-b58b-4df1-ad2b-c47dc60c4303', imageId='357577c1-8d1b-4fad-b470-8e55341e26f6'}), log id: 64dbea74 2017-10-01 17:20:36,027-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler9) [3fab6aa9] -- executeIrsBrokerCommand: calling 'setVolumeDescription', parameters: 2017-10-01 17:20:36,027-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler9) [3fab6aa9] ++ spUUID=aca61eac-ed8a-484b-bbee-3645148f6405 2017-10-01 17:20:36,027-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler9) [3fab6aa9] ++ sdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f 2017-10-01 17:20:36,027-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler9) [3fab6aa9] ++ imageGroupGUID=def9e7db-b58b-4df1-ad2b-c47dc60c4303 2017-10-01 17:20:36,027-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler9) [3fab6aa9] ++ volUUID=357577c1-8d1b-4fad-b470-8e55341e26f6 2017-10-01 17:20:36,027-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler9) [3fab6aa9] ++ description={"Updated":true,"Size":481280,"Last Updated":"Sun Oct 01 17:20:31 EDT 2017","Storage Domains":[{"uuid":"c8653dba-63ec-4447-872c-0ffcc6af9c0f"}],"Disk Description":"OVF_STORE"} 2017-10-01 17:20:36,679-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler9) [3fab6aa9] FINISH, SetVolumeDescriptionVDSCommand, log id: 64dbea74 2017-10-01 17:20:36,685-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStorageDomainCommand] (DefaultQuartzScheduler9) [3fab6aa9] Lock freed to object 'EngineLock:{exclusiveLocks='[c8653dba-63ec-4447-872c-0ffcc6af9c0f=STORAGE]', sharedLocks='[aca61eac-ed8a-484b-bbee-3645148f6405=OVF_UPDATE]'}' 2017-10-01 17:20:37,673-04 INFO [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] (DefaultQuartzScheduler10) [3fab6aa9] Command 'ProcessOvfUpdateForStorageDomain' (id: '2bcf7c01-fd1a-445e-b65e-1924089f464e') waiting on child command id: '8b1db5d0-e304-4985-9d55-ee08708e2440' type:'UploadStream' to complete 2017-10-01 17:20:39,670-04 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler1) [118eca44] Polling and updating Async Tasks: 2 tasks, 2 tasks to poll now 2017-10-01 17:20:40,675-04 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler1) [118eca44] SPMAsyncTask::PollTask: Polling task '2e1e9f29-e4f8-4e2c-86bb-406e9e095bd4' (Parent Command 'UploadStream', Parameters Type 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters') returned status 'finished', result 'success'. 2017-10-01 17:20:40,675-04 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler1) [118eca44] BaseAsyncTask::onTaskEndSuccess: Task '2e1e9f29-e4f8-4e2c-86bb-406e9e095bd4' (Parent Command 'UploadStream', Parameters Type 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters') ended successfully. 2017-10-01 17:20:40,677-04 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler1) [118eca44] CommandAsyncTask::endActionIfNecessary: All tasks of command '21fda0bd-eb44-4a0f-b81d-2e0fc1c492bc' has ended -> executing 'endAction' 2017-10-01 17:20:40,677-04 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler1) [118eca44] CommandAsyncTask::endAction: Ending action for '1' tasks (command ID: '21fda0bd-eb44-4a0f-b81d-2e0fc1c492bc'): calling endAction '. 2017-10-01 17:20:40,677-04 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler1) [118eca44] SPMAsyncTask::PollTask: Polling task '76f839c1-f1d7-487f-8221-d43539c8f103' (Parent Command 'UploadStream', Parameters Type 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters') returned status 'finished', result 'success'. 2017-10-01 17:20:40,677-04 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler1) [118eca44] BaseAsyncTask::onTaskEndSuccess: Task '76f839c1-f1d7-487f-8221-d43539c8f103' (Parent Command 'UploadStream', Parameters Type 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters') ended successfully. 2017-10-01 17:20:40,677-04 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-6-thread-2) [118eca44] CommandAsyncTask::endCommandAction [within thread] context: Attempting to endAction 'UploadStream', 2017-10-01 17:20:40,679-04 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler1) [118eca44] CommandAsyncTask::endActionIfNecessary: All tasks of command '8b1db5d0-e304-4985-9d55-ee08708e2440' has ended -> executing 'endAction' 2017-10-01 17:20:40,679-04 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler1) [118eca44] CommandAsyncTask::endAction: Ending action for '1' tasks (command ID: '8b1db5d0-e304-4985-9d55-ee08708e2440'): calling endAction '. 2017-10-01 17:20:40,679-04 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-6-thread-14) [118eca44] CommandAsyncTask::endCommandAction [within thread] context: Attempting to endAction 'UploadStream', 2017-10-01 17:20:40,682-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.UploadStreamCommand] (org.ovirt.thread.pool-6-thread-2) [3fab6aa9] Ending command 'org.ovirt.engine.core.bll.storage.ovfstore.UploadStreamCommand' successfully. 2017-10-01 17:20:40,683-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.UploadStreamCommand] (org.ovirt.thread.pool-6-thread-14) [3fab6aa9] Ending command 'org.ovirt.engine.core.bll.storage.ovfstore.UploadStreamCommand' successfully. 2017-10-01 17:20:40,686-04 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-6-thread-2) [3fab6aa9] CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type 'UploadStream' completed, handling the result. 2017-10-01 17:20:40,686-04 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-6-thread-2) [3fab6aa9] CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type 'UploadStream' succeeded, clearing tasks. 2017-10-01 17:20:40,686-04 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-6-thread-2) [3fab6aa9] SPMAsyncTask::ClearAsyncTask: Attempting to clear task '2e1e9f29-e4f8-4e2c-86bb-406e9e095bd4' 2017-10-01 17:20:40,687-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-6-thread-2) [3fab6aa9] START, SPMClearTaskVDSCommand( SPMTaskGuidBaseVDSCommandParameters:{runAsync='true', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', ignoreFailoverLimit='false', taskId='2e1e9f29-e4f8-4e2c-86bb-406e9e095bd4'}), log id: 32128220 2017-10-01 17:20:40,687-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-6-thread-2) [3fab6aa9] START, HSMClearTaskVDSCommand(HostName = virt1, HSMTaskGuidBaseVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', taskId='2e1e9f29-e4f8-4e2c-86bb-406e9e095bd4'}), log id: 4bbdd157 2017-10-01 17:20:40,689-04 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-6-thread-14) [3fab6aa9] CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type 'UploadStream' completed, handling the result. 2017-10-01 17:20:40,690-04 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-6-thread-14) [3fab6aa9] CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type 'UploadStream' succeeded, clearing tasks. 2017-10-01 17:20:40,690-04 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-6-thread-14) [3fab6aa9] SPMAsyncTask::ClearAsyncTask: Attempting to clear task '76f839c1-f1d7-487f-8221-d43539c8f103' 2017-10-01 17:20:40,690-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-6-thread-14) [3fab6aa9] START, SPMClearTaskVDSCommand( SPMTaskGuidBaseVDSCommandParameters:{runAsync='true', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', ignoreFailoverLimit='false', taskId='76f839c1-f1d7-487f-8221-d43539c8f103'}), log id: 4a3038a0 2017-10-01 17:20:41,698-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-6-thread-2) [3fab6aa9] FINISH, HSMClearTaskVDSCommand, log id: 4bbdd157 2017-10-01 17:20:41,698-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-6-thread-2) [3fab6aa9] FINISH, SPMClearTaskVDSCommand, log id: 32128220 2017-10-01 17:20:41,698-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-6-thread-14) [3fab6aa9] START, HSMClearTaskVDSCommand(HostName = virt1, HSMTaskGuidBaseVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', taskId='76f839c1-f1d7-487f-8221-d43539c8f103'}), log id: 630bf602 2017-10-01 17:20:41,702-04 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-6-thread-2) [3fab6aa9] BaseAsyncTask::removeTaskFromDB: Removed task '2e1e9f29-e4f8-4e2c-86bb-406e9e095bd4' from DataBase 2017-10-01 17:20:41,702-04 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-6-thread-2) [3fab6aa9] CommandAsyncTask::HandleEndActionResult [within thread]: Removing CommandMultiAsyncTasks object for entity '21fda0bd-eb44-4a0f-b81d-2e0fc1c492bc' 2017-10-01 17:20:42,710-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-6-thread-14) [3fab6aa9] FINISH, HSMClearTaskVDSCommand, log id: 630bf602 2017-10-01 17:20:42,710-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-6-thread-14) [3fab6aa9] FINISH, SPMClearTaskVDSCommand, log id: 4a3038a0 2017-10-01 17:20:42,712-04 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-6-thread-14) [3fab6aa9] BaseAsyncTask::removeTaskFromDB: Removed task '76f839c1-f1d7-487f-8221-d43539c8f103' from DataBase 2017-10-01 17:20:42,712-04 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-6-thread-14) [3fab6aa9] CommandAsyncTask::HandleEndActionResult [within thread]: Removing CommandMultiAsyncTasks object for entity '8b1db5d0-e304-4985-9d55-ee08708e2440' 2017-10-01 17:20:45,692-04 INFO [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] (DefaultQuartzScheduler6) [3fab6aa9] Command 'ProcessOvfUpdateForStorageDomain' id: '2bcf7c01-fd1a-445e-b65e-1924089f464e' child commands '[8b1db5d0-e304-4985-9d55-ee08708e2440, 21fda0bd-eb44-4a0f-b81d-2e0fc1c492bc]' executions were completed, status 'SUCCEEDED' 2017-10-01 17:20:46,702-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStorageDomainCommand] (DefaultQuartzScheduler10) [3fab6aa9] Ending command 'org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStorageDomainCommand' successfully. 2017-10-01 17:20:57,991-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [51d042cf] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 17:20:58,007-04 INFO [org.ovirt.engine.core.sso.servlets.OAuthRevokeServlet] (default task-45) [] User jas@eecs.yorku.ca successfully logged out 2017-10-01 17:20:58,020-04 INFO [org.ovirt.engine.core.bll.aaa.TerminateSessionsForTokenCommand] (default task-47) [29b410c6] Running command: TerminateSessionsForTokenCommand internal: true. 2017-10-01 17:21:58,045-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [6413fdd8] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 17:21:59,220-04 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler6) [2c1e3a7] Setting new tasks map. The map contains now 0 tasks 2017-10-01 17:21:59,220-04 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler6) [2c1e3a7] Cleared all tasks of pool 'aca61eac-ed8a-484b-bbee-3645148f6405'. 2017-10-01 17:22:58,070-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [305a9b3c] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 17:23:58,089-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [28a0b023] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 17:24:58,110-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [7f12b6bd] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 17:25:58,135-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [7d91670e] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 17:26:58,154-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [3ba68796] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 17:27:58,179-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [32ec1b2b] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 17:28:29,110-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-6-thread-7) [4fd7e41f] START, GetFileStatsVDSCommand( GetFileStatsParameters:{runAsync='true', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', ignoreFailoverLimit='false'}), log id: 5c5b60e0 2017-10-01 17:28:30,123-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-6-thread-7) [4fd7e41f] FINISH, GetFileStatsVDSCommand, return: {virtio-win-0.1.126.iso={status=0, ctime=1497024605.0, size=155856896}, server2012r2_virtio.iso={status=0, ctime=1497297085.0, size=4977590272}, en_windows_7_professional_with_sp1_x64_dvd_621750.iso={status=0, ctime=1435288543.0, size=3319478272}, virtio-win-0.1.105.iso={status=0, ctime=1435288648.0, size=56584192}}, log id: 5c5b60e0 2017-10-01 17:28:30,126-04 INFO [org.ovirt.engine.core.bll.storage.domain.IsoDomainListSynchronizer] (org.ovirt.thread.pool-6-thread-7) [4fd7e41f] Finished automatic refresh process for 'ISO' file type with success, for storage domain id '63d680c7-4c94-4b29-a6cb-f288a128b9bc'. 2017-10-01 17:28:58,198-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [2de5f645] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 17:29:58,216-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [32fda966] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 17:30:58,236-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [26e8c1aa] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 17:31:58,256-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [13e5f2dc] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 17:32:58,282-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [7161ce71] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 17:33:58,304-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [2a36a452] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 17:34:58,323-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [718b025a] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 17:35:58,344-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [c503ebe] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 17:36:58,363-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [783cdb8c] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 17:37:58,387-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [238aeba5] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 17:38:58,407-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [3e26d813] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 17:39:58,426-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [410d01fc] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 17:40:58,445-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [4491de08] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 17:41:58,466-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [632d285a] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 17:42:58,489-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [273e41ff] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 17:43:58,508-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [4e2a0a7b] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 17:44:58,526-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [290dbf14] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 17:45:58,545-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [6e371588] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 17:46:58,563-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [2b14bb61] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 17:47:58,588-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [4a6554cc] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 17:48:58,608-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler7) [44d0ba17] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 17:49:58,628-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [7f4c390c] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 17:50:58,651-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [dde174f] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 17:51:58,671-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [3d3ba612] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 17:52:58,697-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [7c0113b5] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 17:53:58,716-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [fb370bc] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 17:54:58,732-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [728b22ed] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 17:55:58,749-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [15b79297] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 17:56:58,770-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [471598fc] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 17:57:58,794-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [c39c252] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 17:58:58,816-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [3a34a346] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 17:59:58,837-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [7e618de8] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 18:00:58,857-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [6544fbe7] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 18:01:58,877-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [33e2f7dd] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 18:02:58,902-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [7c7c318e] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 18:03:58,921-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [5fe9bfbf] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 18:04:58,941-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [23f5e11] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 18:05:58,961-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [888eeee] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 18:06:58,983-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [697a9092] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 18:07:59,008-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [35a2299b] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 18:08:59,027-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [5947f906] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 18:09:59,049-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [26d9a8fc] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 18:10:59,068-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [25722e9] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 18:11:59,087-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [52e59f1a] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 18:12:59,114-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [476eeb80] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 18:13:59,135-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [639abb68] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 18:14:59,153-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [5e4d939] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 18:15:59,172-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [43a6dcbb] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 18:16:59,192-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler7) [69d22b70] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 18:17:59,216-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [4caf6ca] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 18:18:59,237-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler7) [3b42d2df] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 18:19:20,461-04 INFO [org.ovirt.engine.core.bll.EngineBackupAwarenessManager] (DefaultQuartzScheduler9) [2cc82f4f] Backup check started. 2017-10-01 18:19:20,464-04 INFO [org.ovirt.engine.core.bll.EngineBackupAwarenessManager] (DefaultQuartzScheduler9) [2cc82f4f] Backup check completed. 2017-10-01 18:19:59,257-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [29643ef5] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 18:20:36,686-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.OvfDataUpdater] (DefaultQuartzScheduler4) [5e5dfdf0] Attempting to update VMs/Templates Ovf. 2017-10-01 18:20:36,689-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler4) [21e3338f] Before acquiring and wait lock 'EngineLock:{exclusiveLocks='[aca61eac-ed8a-484b-bbee-3645148f6405=OVF_UPDATE]', sharedLocks=''}' 2017-10-01 18:20:36,689-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler4) [21e3338f] Lock-wait acquired to object 'EngineLock:{exclusiveLocks='[aca61eac-ed8a-484b-bbee-3645148f6405=OVF_UPDATE]', sharedLocks=''}' 2017-10-01 18:20:36,690-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler4) [21e3338f] Running command: ProcessOvfUpdateForStoragePoolCommand internal: true. Entities affected : ID: aca61eac-ed8a-484b-bbee-3645148f6405 Type: StoragePool 2017-10-01 18:20:36,695-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler4) [21e3338f] Attempting to update VM OVFs in Data Center 'EECS-Primary' 2017-10-01 18:20:36,701-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler4) [21e3338f] Successfully updated VM OVFs in Data Center 'EECS-Primary' 2017-10-01 18:20:36,701-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler4) [21e3338f] Attempting to update template OVFs in Data Center 'EECS-Primary' 2017-10-01 18:20:36,702-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler4) [21e3338f] Successfully updated templates OVFs in Data Center 'EECS-Primary' 2017-10-01 18:20:36,702-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler4) [21e3338f] Attempting to remove unneeded template/vm OVFs in Data Center 'EECS-Primary' 2017-10-01 18:20:36,704-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler4) [21e3338f] Successfully removed unneeded template/vm OVFs in Data Center 'EECS-Primary' 2017-10-01 18:20:36,705-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler4) [21e3338f] Lock freed to object 'EngineLock:{exclusiveLocks='[aca61eac-ed8a-484b-bbee-3645148f6405=OVF_UPDATE]', sharedLocks=''}' 2017-10-01 18:20:59,276-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [6e6cf035] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 18:21:59,293-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [503b6bf5] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 18:22:59,316-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [bcd7f68] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 18:23:59,335-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [219365ca] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 18:24:59,355-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [73065c81] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 18:25:59,374-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [7282b765] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 18:26:59,394-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [28818044] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 18:27:59,420-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler7) [35b03597] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 18:28:30,130-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-6-thread-24) [1a998dfd] START, GetFileStatsVDSCommand( GetFileStatsParameters:{runAsync='true', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', ignoreFailoverLimit='false'}), log id: 6f83aa89 2017-10-01 18:28:30,736-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-6-thread-24) [1a998dfd] FINISH, GetFileStatsVDSCommand, return: {virtio-win-0.1.126.iso={status=0, ctime=1497024605.0, size=155856896}, server2012r2_virtio.iso={status=0, ctime=1497297085.0, size=4977590272}, en_windows_7_professional_with_sp1_x64_dvd_621750.iso={status=0, ctime=1435288543.0, size=3319478272}, virtio-win-0.1.105.iso={status=0, ctime=1435288648.0, size=56584192}}, log id: 6f83aa89 2017-10-01 18:28:30,740-04 INFO [org.ovirt.engine.core.bll.storage.domain.IsoDomainListSynchronizer] (org.ovirt.thread.pool-6-thread-24) [1a998dfd] Finished automatic refresh process for 'ISO' file type with success, for storage domain id '63d680c7-4c94-4b29-a6cb-f288a128b9bc'. 2017-10-01 18:28:59,439-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [2fddc774] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 18:29:59,463-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [268e644c] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 18:30:59,481-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [3f5527df] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 18:31:59,499-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [21e0eeee] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 18:32:59,523-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [2e56496d] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 18:33:59,542-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler7) [1c6b2be] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 18:34:59,561-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [7a3ed458] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 18:35:59,582-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [583e283a] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 18:36:59,601-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler7) [4e1fd61e] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 18:37:59,625-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [33f59ad2] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 18:38:59,644-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [11eee9c4] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 18:39:59,664-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [77cd0d07] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 18:40:59,685-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [47ef62a5] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 18:41:59,703-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [784483d9] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 18:42:59,728-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [50ac4d18] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 18:43:59,752-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [6b0e5222] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 18:44:59,772-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [55680159] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 18:45:59,791-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler7) [30beff2e] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 18:46:59,810-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [14db7e88] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 18:47:59,839-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [688c69d6] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 18:48:59,859-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [6901ec74] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 18:49:59,878-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [75b1f814] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 18:50:59,896-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [4debca8c] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 18:51:59,912-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [38222f9d] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 18:52:59,934-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [367b0fd6] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 18:53:59,954-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [42a0bb16] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 18:54:59,973-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [64d5e83c] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 18:55:59,993-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [385610e1] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 18:57:00,012-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [4ddd25ee] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 18:58:00,039-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [39746ff8] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 18:59:00,058-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [18065ac3] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 19:00:00,077-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [356f9e7e] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 19:01:00,101-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [625c04c0] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 19:02:00,121-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [3c38518b] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 19:03:00,145-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [2add63a0] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 19:04:00,164-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler7) [4f4e61c3] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 19:05:00,184-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [6f9e1fd8] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 19:06:00,204-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [4acdcf75] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 19:07:00,223-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [fb2621] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 19:08:00,249-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [2ddf0216] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 19:09:00,268-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [61a588be] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 19:10:00,288-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [7b5275d] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 19:11:00,308-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [4e207e83] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 19:12:00,327-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [458b2df] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 19:13:00,351-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [13da740a] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 19:14:00,372-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [513e856f] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 19:15:00,396-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [75f8eb7a] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 19:16:00,416-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [6522e146] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 19:17:00,436-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [25dada0a] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 19:18:00,462-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler7) [2bbdfe17] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 19:19:00,481-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [1aee2133] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 19:20:00,500-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [5017b04f] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 19:20:36,706-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.OvfDataUpdater] (DefaultQuartzScheduler7) [ac163c5] Attempting to update VMs/Templates Ovf. 2017-10-01 19:20:36,709-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler7) [2873c0aa] Before acquiring and wait lock 'EngineLock:{exclusiveLocks='[aca61eac-ed8a-484b-bbee-3645148f6405=OVF_UPDATE]', sharedLocks=''}' 2017-10-01 19:20:36,709-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler7) [2873c0aa] Lock-wait acquired to object 'EngineLock:{exclusiveLocks='[aca61eac-ed8a-484b-bbee-3645148f6405=OVF_UPDATE]', sharedLocks=''}' 2017-10-01 19:20:36,710-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler7) [2873c0aa] Running command: ProcessOvfUpdateForStoragePoolCommand internal: true. Entities affected : ID: aca61eac-ed8a-484b-bbee-3645148f6405 Type: StoragePool 2017-10-01 19:20:36,715-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler7) [2873c0aa] Attempting to update VM OVFs in Data Center 'EECS-Primary' 2017-10-01 19:20:36,720-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler7) [2873c0aa] Successfully updated VM OVFs in Data Center 'EECS-Primary' 2017-10-01 19:20:36,720-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler7) [2873c0aa] Attempting to update template OVFs in Data Center 'EECS-Primary' 2017-10-01 19:20:36,721-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler7) [2873c0aa] Successfully updated templates OVFs in Data Center 'EECS-Primary' 2017-10-01 19:20:36,721-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler7) [2873c0aa] Attempting to remove unneeded template/vm OVFs in Data Center 'EECS-Primary' 2017-10-01 19:20:36,722-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler7) [2873c0aa] Successfully removed unneeded template/vm OVFs in Data Center 'EECS-Primary' 2017-10-01 19:20:36,723-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler7) [2873c0aa] Lock freed to object 'EngineLock:{exclusiveLocks='[aca61eac-ed8a-484b-bbee-3645148f6405=OVF_UPDATE]', sharedLocks=''}' 2017-10-01 19:21:00,519-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [5b7ebd1f] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 19:22:00,536-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [390daee2] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 19:23:00,562-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [3b3c4da] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 19:24:00,581-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [78df7556] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 19:25:00,600-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [24cf6ad0] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 19:26:00,619-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [7652e69a] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 19:27:00,639-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [4026cfa] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 19:28:00,663-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [7ffe33ca] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 19:28:30,746-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-6-thread-44) [17dc49ac] START, GetFileStatsVDSCommand( GetFileStatsParameters:{runAsync='true', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', ignoreFailoverLimit='false'}), log id: 11dc17fb 2017-10-01 19:28:32,428-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-6-thread-44) [17dc49ac] FINISH, GetFileStatsVDSCommand, return: {virtio-win-0.1.126.iso={status=0, ctime=1497024605.0, size=155856896}, server2012r2_virtio.iso={status=0, ctime=1497297085.0, size=4977590272}, en_windows_7_professional_with_sp1_x64_dvd_621750.iso={status=0, ctime=1435288543.0, size=3319478272}, virtio-win-0.1.105.iso={status=0, ctime=1435288648.0, size=56584192}}, log id: 11dc17fb 2017-10-01 19:28:32,431-04 INFO [org.ovirt.engine.core.bll.storage.domain.IsoDomainListSynchronizer] (org.ovirt.thread.pool-6-thread-44) [17dc49ac] Finished automatic refresh process for 'ISO' file type with success, for storage domain id '63d680c7-4c94-4b29-a6cb-f288a128b9bc'. 2017-10-01 19:29:00,682-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [672e0933] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 19:30:00,703-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [156a1e49] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 19:31:00,723-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [55cec35a] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 19:32:00,745-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler7) [25de2759] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 19:33:00,769-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [5527338] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 19:34:00,788-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [269f5cee] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 19:35:00,807-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [771286b3] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 19:36:00,826-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [568ddebb] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 19:37:00,846-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [5222edff] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 19:38:00,868-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [6fd01396] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 19:39:00,886-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [31bfd596] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 19:40:00,905-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [36e4a0f5] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 19:41:00,925-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [2c5d2a89] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 19:42:00,944-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [41b50a1b] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 19:43:00,967-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [78a5a38f] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 19:44:00,985-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [1b3fdb3a] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 19:45:01,004-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler7) [68c4bf61] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 19:46:01,023-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [7bf86b03] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 19:47:01,042-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [4fd375d3] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 19:48:01,068-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [39fc49b0] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 19:49:01,090-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [13886d47] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 19:50:01,109-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [2a9568e2] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 19:51:01,131-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [17beb701] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 19:52:01,150-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [372eb8bd] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 19:53:01,173-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [b36dbea] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 19:54:01,190-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [57e6ed0b] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 19:55:01,211-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [53388c21] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 19:56:01,231-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [736469eb] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 19:57:01,252-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [58ac1614] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 19:58:01,278-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [4f63109] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 19:59:01,297-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [29f9705a] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 20:00:01,314-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [404c53f5] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 20:01:01,335-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [2dc2908e] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 20:02:01,357-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [34839a86] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 20:03:01,386-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [6013cc82] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 20:04:01,402-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [3c2968a7] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 20:05:01,420-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [34c897d] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 20:06:01,441-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [5777e6d3] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 20:07:01,460-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [48895ab1] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 20:08:01,563-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [7448f8f2] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 20:09:01,582-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [1c20c2b1] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 20:10:01,598-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [1b20822d] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 20:11:01,620-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler7) [3d16eab6] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 20:12:01,640-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [3fe70af4] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 20:13:01,662-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [16c53d7e] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 20:14:01,681-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [3d644b49] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 20:15:01,701-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [b4ad17] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 20:16:01,718-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [124b8b03] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 20:17:01,737-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [204c56cd] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 20:18:01,761-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [3bfd2a4c] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 20:19:01,782-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [673c04d3] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 20:20:01,804-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [2d044b28] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 20:20:36,724-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.OvfDataUpdater] (DefaultQuartzScheduler3) [11673535] Attempting to update VMs/Templates Ovf. 2017-10-01 20:20:36,727-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler3) [79ac9d75] Before acquiring and wait lock 'EngineLock:{exclusiveLocks='[aca61eac-ed8a-484b-bbee-3645148f6405=OVF_UPDATE]', sharedLocks=''}' 2017-10-01 20:20:36,728-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler3) [79ac9d75] Lock-wait acquired to object 'EngineLock:{exclusiveLocks='[aca61eac-ed8a-484b-bbee-3645148f6405=OVF_UPDATE]', sharedLocks=''}' 2017-10-01 20:20:36,729-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler3) [79ac9d75] Running command: ProcessOvfUpdateForStoragePoolCommand internal: true. Entities affected : ID: aca61eac-ed8a-484b-bbee-3645148f6405 Type: StoragePool 2017-10-01 20:20:36,793-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler3) [79ac9d75] Attempting to update VM OVFs in Data Center 'EECS-Primary' 2017-10-01 20:20:36,798-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler3) [79ac9d75] Successfully updated VM OVFs in Data Center 'EECS-Primary' 2017-10-01 20:20:36,798-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler3) [79ac9d75] Attempting to update template OVFs in Data Center 'EECS-Primary' 2017-10-01 20:20:36,799-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler3) [79ac9d75] Successfully updated templates OVFs in Data Center 'EECS-Primary' 2017-10-01 20:20:36,799-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler3) [79ac9d75] Attempting to remove unneeded template/vm OVFs in Data Center 'EECS-Primary' 2017-10-01 20:20:36,801-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler3) [79ac9d75] Successfully removed unneeded template/vm OVFs in Data Center 'EECS-Primary' 2017-10-01 20:20:36,802-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler3) [79ac9d75] Lock freed to object 'EngineLock:{exclusiveLocks='[aca61eac-ed8a-484b-bbee-3645148f6405=OVF_UPDATE]', sharedLocks=''}' 2017-10-01 20:21:01,825-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [3658c3e7] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 20:22:01,845-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [6b5b60a9] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 20:23:01,869-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [6c24bdc] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 20:24:01,890-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [74afdd6] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 20:25:01,913-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [277b520] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 20:26:01,932-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler7) [6254cf15] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 20:27:01,951-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [171130b3] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 20:28:01,976-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [5eab297d] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 20:28:32,435-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-6-thread-48) [59bca785] START, GetFileStatsVDSCommand( GetFileStatsParameters:{runAsync='true', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', ignoreFailoverLimit='false'}), log id: 6afab0dd 2017-10-01 20:28:32,744-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-6-thread-48) [59bca785] FINISH, GetFileStatsVDSCommand, return: {virtio-win-0.1.126.iso={status=0, ctime=1497024605.0, size=155856896}, server2012r2_virtio.iso={status=0, ctime=1497297085.0, size=4977590272}, en_windows_7_professional_with_sp1_x64_dvd_621750.iso={status=0, ctime=1435288543.0, size=3319478272}, virtio-win-0.1.105.iso={status=0, ctime=1435288648.0, size=56584192}}, log id: 6afab0dd 2017-10-01 20:28:32,749-04 INFO [org.ovirt.engine.core.bll.storage.domain.IsoDomainListSynchronizer] (org.ovirt.thread.pool-6-thread-48) [59bca785] Finished automatic refresh process for 'ISO' file type with success, for storage domain id '63d680c7-4c94-4b29-a6cb-f288a128b9bc'. 2017-10-01 20:29:01,994-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [185582fb] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 20:30:02,015-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [3527925c] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 20:30:27,136-04 INFO [org.ovirt.engine.core.sso.utils.AuthenticationUtils] (default task-348) [] User jas@eecs.yorku.ca successfully logged in with scopes: ovirt-app-admin ovirt-app-api ovirt-app-portal ovirt-ext=auth:sequence-priority=~ ovirt-ext=revoke:revoke-all ovirt-ext=token-info:authz-search ovirt-ext=token-info:public-authz-search ovirt-ext=token-info:validate ovirt-ext=token:password-access 2017-10-01 20:30:27,233-04 INFO [org.ovirt.engine.core.bll.aaa.CreateUserSessionCommand] (default task-345) [308b74c7] Running command: CreateUserSessionCommand internal: false. 2017-10-01 20:30:27,242-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-345) [308b74c7] EVENT_ID: USER_VDC_LOGIN(30), Correlation ID: 308b74c7, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: User jas@eecs.yorku.ca-authz logged in. 2017-10-01 20:30:30,478-04 ERROR [org.ovirt.engine.core.utils.servlet.ServletUtils] (default task-364) [] Can't read file '/usr/share/ovirt-engine/files/spice/SpiceVersion.txt' for request '/ovirt-engine/services/files/spice/SpiceVersion.txt', will send a 404 error response. 2017-10-01 20:30:58,829-04 INFO [org.ovirt.engine.core.bll.SetVmTicketCommand] (default task-366) [3ebc711d] Running command: SetVmTicketCommand internal: false. Entities affected : ID: 1734fed5-fc97-4714-bb63-55e8f4eb6d34 Type: VMAction group CONNECT_TO_VM with role type USER 2017-10-01 20:30:58,858-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (default task-366) [3ebc711d] START, SetVmTicketVDSCommand(HostName = virt2, SetVmTicketVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vmId='1734fed5-fc97-4714-bb63-55e8f4eb6d34', protocol='SPICE', ticket='HHddK+SsJDIo', validTime='120', userName='jas', userId='4a225b32-bc9d-42b7-90f8-f7848db0e59f', disconnectAction='LOCK_SCREEN'}), log id: 1fdff0e6 2017-10-01 20:30:59,057-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (default task-366) [3ebc711d] FINISH, SetVmTicketVDSCommand, log id: 1fdff0e6 2017-10-01 20:30:59,066-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-366) [3ebc711d] EVENT_ID: VM_SET_TICKET(164), Correlation ID: 3ebc711d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: User jas@eecs.yorku.ca-authz initiated console session for VM wiki 2017-10-01 20:31:02,032-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [285ceaac] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 20:31:10,864-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [1864ac36] EVENT_ID: VM_CONSOLE_CONNECTED(167), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: User jas@eecs.yorku.ca-authz is connected to VM wiki. 2017-10-01 20:31:25,891-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler4) [1a3cb9dc] VM '1734fed5-fc97-4714-bb63-55e8f4eb6d34' was reported as Down on VDS 'a0ccfd11-3402-4eba-a119-4987561b738e'(virt2) 2017-10-01 20:31:25,891-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (DefaultQuartzScheduler4) [1a3cb9dc] START, DestroyVDSCommand(HostName = virt2, DestroyVmVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vmId='1734fed5-fc97-4714-bb63-55e8f4eb6d34', force='false', secondsToWait='0', gracefully='false', reason='', ignoreNoVm='true'}), log id: 408d779 2017-10-01 20:31:25,900-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (DefaultQuartzScheduler4) [1a3cb9dc] FINISH, DestroyVDSCommand, log id: 408d779 2017-10-01 20:31:25,900-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler4) [1a3cb9dc] VM '1734fed5-fc97-4714-bb63-55e8f4eb6d34'(wiki) moved from 'Up' --> 'Down' 2017-10-01 20:31:25,973-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [1a3cb9dc] EVENT_ID: VM_DOWN(61), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM wiki is down. Exit message: User shut down from within the guest 2017-10-01 20:31:25,986-04 INFO [org.ovirt.engine.core.bll.ProcessDownVmCommand] (org.ovirt.thread.pool-6-thread-2) [353bd562] Running command: ProcessDownVmCommand internal: true. 2017-10-01 20:31:40,988-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmsStatisticsFetcher] (DefaultQuartzScheduler8) [68931c4] Fetched 5 VMs from VDS 'a0ccfd11-3402-4eba-a119-4987561b738e' 2017-10-01 20:32:02,051-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler7) [2117528f] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 20:33:02,075-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [72fbc261] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 20:34:02,094-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [669d6954] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 20:35:02,113-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [4cb43e15] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 20:35:22,963-04 INFO [org.ovirt.engine.core.bll.RunVmCommand] (default task-33) [30fa922c-63b2-4b67-b376-10029722eb4a] Lock Acquired to object 'EngineLock:{exclusiveLocks='[1734fed5-fc97-4714-bb63-55e8f4eb6d34=VM]', sharedLocks=''}' 2017-10-01 20:35:23,039-04 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (default task-33) [30fa922c-63b2-4b67-b376-10029722eb4a] START, IsVmDuringInitiatingVDSCommand( IsVmDuringInitiatingVDSCommandParameters:{runAsync='true', vmId='1734fed5-fc97-4714-bb63-55e8f4eb6d34'}), log id: 7a51d39c 2017-10-01 20:35:23,039-04 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (default task-33) [30fa922c-63b2-4b67-b376-10029722eb4a] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 7a51d39c 2017-10-01 20:35:23,136-04 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-6-thread-10) [30fa922c-63b2-4b67-b376-10029722eb4a] Running command: RunVmCommand internal: false. Entities affected : ID: 1734fed5-fc97-4714-bb63-55e8f4eb6d34 Type: VMAction group RUN_VM with role type USER 2017-10-01 20:35:23,221-04 INFO [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (org.ovirt.thread.pool-6-thread-10) [30fa922c-63b2-4b67-b376-10029722eb4a] START, UpdateVmDynamicDataVDSCommand( UpdateVmDynamicDataVDSCommandParameters:{runAsync='true', hostId='null', vmId='1734fed5-fc97-4714-bb63-55e8f4eb6d34', vmDynamic='org.ovirt.engine.core.common.businessentities.VmDynamic@386199f6'}), log id: 18ad96ef 2017-10-01 20:35:23,223-04 INFO [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (org.ovirt.thread.pool-6-thread-10) [30fa922c-63b2-4b67-b376-10029722eb4a] FINISH, UpdateVmDynamicDataVDSCommand, log id: 18ad96ef 2017-10-01 20:35:23,227-04 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-6-thread-10) [30fa922c-63b2-4b67-b376-10029722eb4a] START, CreateVmVDSCommand( CreateVmVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vmId='1734fed5-fc97-4714-bb63-55e8f4eb6d34', vm='VM [wiki]'}), log id: 739b1abd 2017-10-01 20:35:23,231-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-6-thread-10) [30fa922c-63b2-4b67-b376-10029722eb4a] START, CreateVDSCommand(HostName = virt2, CreateVmVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vmId='1734fed5-fc97-4714-bb63-55e8f4eb6d34', vm='VM [wiki]'}), log id: 6d095259 2017-10-01 20:35:23,234-04 INFO [org.ovirt.engine.core.vdsbroker.builder.vminfo.VmInfoBuilderImpl] (org.ovirt.thread.pool-6-thread-10) [30fa922c-63b2-4b67-b376-10029722eb4a] Bootable disk '4096159b-990c-49d0-81fb-4cd934f00c47' set to index '0' 2017-10-01 20:35:23,248-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-6-thread-10) [30fa922c-63b2-4b67-b376-10029722eb4a] VM {memGuaranteedSize=4096, smpThreadsPerCore=1, cpuType=SandyBridge, vmId=1734fed5-fc97-4714-bb63-55e8f4eb6d34, acpiEnable=true, vmType=kvm, smp=2, smpCoresPerSocket=1, emulatedMachine=pc-i440fx-rhel7.3.0, smartcardEnable=false, guestNumaNodes=[{memory=4096, cpus=0,1, nodeIndex=0}], transparentHugePages=true, displayNetwork=yorku, vmName=wiki, maxVCpus=16, kvmEnable=true, devices=[{address={bus=0x00, domain=0x0000, function=0x0, slot=0x02, type=pci}, type=video, specParams={heads=1, vram=32768}, device=qxl, deviceId=ccde18db-7091-4bc3-bbee-a8fab06696d5}, {type=graphics, specParams={fileTransferEnable=true, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,ssmartcard,susbredir, copyPasteEnable=true}, device=spice, deviceId=0000002d-002d-002d-002d-0000000001ac}, {iface=ide, shared=false, path=, address={bus=1, controller=0, unit=0, type=drive, target=0}, readonly=true, index=2, type=disk, specParams={path=}, device=cdrom, deviceId=9f75bd5a-318a-41ae-8cad-dcf457a2a1ed}, {discard=false, shared=false, address={bus=0x00, domain=0x0000, function=0x0, slot=0x07, type=pci}, imageID=4096159b-990c-49d0-81fb-4cd934f00c47, format=raw, index=0, optional=false, type=disk, deviceId=4096159b-990c-49d0-81fb-4cd934f00c47, domainID=c8653dba-63ec-4447-872c-0ffcc6af9c0f, propagateErrors=off, iface=virtio, readonly=false, bootOrder=1, poolID=aca61eac-ed8a-484b-bbee-3645148f6405, volumeID=e58de177-2cfb-42fe-8884-2de3794056db, specParams={}, device=disk}, {discard=false, shared=false, address={bus=0x00, domain=0x0000, function=0x0, slot=0x08, type=pci}, imageID=fafeface-d302-4c3f-97b8-941b6bc3545f, format=raw, optional=false, type=disk, deviceId=fafeface-d302-4c3f-97b8-941b6bc3545f, domainID=c8653dba-63ec-4447-872c-0ffcc6af9c0f, propagateErrors=off, iface=virtio, readonly=false, poolID=aca61eac-ed8a-484b-bbee-3645148f6405, volumeID=53fc926a-8037-47b9-b7ee-da86132a4c78, specParams={}, device=disk}, {filter=vdsm-no-mac-spoofing, nicModel=pv, address={bus=0x00, domain=0x0000, function=0x0, slot=0x03, type=pci}, type=interface, specParams={inbound={}, outbound={}}, device=bridge, linkActive=true, deviceId=f88584d1-8f34-4bd6-a4b5-def75771b329, macAddr=00:1a:4a:32:3d:07, network=yorku}, {filter=vdsm-no-mac-spoofing, nicModel=pv, address={bus=0x00, domain=0x0000, function=0x0, slot=0x04, type=pci}, type=interface, specParams={inbound={}, outbound={}}, device=bridge, linkActive=true, deviceId=a3f3c6f5-2862-42bf-ad3e-305089af1034, macAddr=00:1a:4a:32:3d:08, network=mrpriv}, {address={bus=0x00, domain=0x0000, function=0x2, slot=0x01, type=pci}, index=0, model=piix3-uhci, type=controller, specParams={}, device=usb, deviceId=62494988-ca9d-4d45-ac6b-31b07c4de816}, {address={bus=0x00, domain=0x0000, function=0x0, slot=0x05, type=pci}, index=0, model=virtio-scsi, type=controller, specParams={}, device=scsi, deviceId=81b8187a-8ff3-4f9d-b989-7cb05efd5205}, {address={bus=0x00, domain=0x0000, function=0x0, slot=0x06, type=pci}, type=controller, specParams={}, device=virtio-serial, deviceId=24498793-8cfa-4f4c-be94-bbc0ef12f88f}, {model=virtio, type=rng, specParams={source=urandom}, device=virtio, deviceId=ae6c7e6e-b9d4-4b03-8593-1f0e6adbf707}], custom={device_0a5bd34d-c914-4d64-b6ac-936668020d32device_859f9cf5-51b7-4607-98a6-e089825ca491device_2bf2471b-8433-4358-98cb-9d827615a722=VmDevice:{id='VmDeviceId:{deviceId='2bf2471b-8433-4358-98cb-9d827615a722', vmId='1734fed5-fc97-4714-bb63-55e8f4eb6d34'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=2}', managed='false', plugged='true', readOnly='false', deviceAlias='channel1', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_0a5bd34d-c914-4d64-b6ac-936668020d32=VmDevice:{id='VmDeviceId:{deviceId='0a5bd34d-c914-4d64-b6ac-936668020d32', vmId='1734fed5-fc97-4714-bb63-55e8f4eb6d34'}', device='ide', type='CONTROLLER', bootOrder='0', specParams='[]', address='{slot=0x01, bus=0x00, domain=0x0000, type=pci, function=0x1}', managed='false', plugged='true', readOnly='false', deviceAlias='ide', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_0a5bd34d-c914-4d64-b6ac-936668020d32device_859f9cf5-51b7-4607-98a6-e089825ca491=VmDevice:{id='VmDeviceId:{deviceId='859f9cf5-51b7-4607-98a6-e089825ca491', vmId='1734fed5-fc97-4714-bb63-55e8f4eb6d34'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=1}', managed='false', plugged='true', readOnly='false', deviceAlias='channel0', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_0a5bd34d-c914-4d64-b6ac-936668020d32device_859f9cf5-51b7-4607-98a6-e089825ca491device_2bf2471b-8433-4358-98cb-9d827615a722device_2aa10df5-bab4-405a-9909-b2d560c41883=VmDevice:{id='VmDeviceId:{deviceId='2aa10df5-bab4-405a-9909-b2d560c41883', vmId='1734fed5-fc97-4714-bb63-55e8f4eb6d34'}', device='spicevmc', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=3}', managed='false', plugged='true', readOnly='false', deviceAlias='channel2', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}}, display=qxl, timeOffset=0, nice=0, maxMemSize=16384, maxMemSlots=16, bootMenuEnable=false, memSize=4096} 2017-10-01 20:35:23,952-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-6-thread-10) [30fa922c-63b2-4b67-b376-10029722eb4a] FINISH, CreateVDSCommand, log id: 6d095259 2017-10-01 20:35:23,956-04 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-6-thread-10) [30fa922c-63b2-4b67-b376-10029722eb4a] FINISH, CreateVmVDSCommand, return: WaitForLaunch, log id: 739b1abd 2017-10-01 20:35:23,957-04 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-6-thread-10) [30fa922c-63b2-4b67-b376-10029722eb4a] Lock freed to object 'EngineLock:{exclusiveLocks='[1734fed5-fc97-4714-bb63-55e8f4eb6d34=VM]', sharedLocks=''}' 2017-10-01 20:35:23,970-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-10) [30fa922c-63b2-4b67-b376-10029722eb4a] EVENT_ID: USER_STARTED_VM(153), Correlation ID: 30fa922c-63b2-4b67-b376-10029722eb4a, Job ID: ba54f4e9-1e79-4f20-9967-00163a5a958a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM wiki was started by jas@eecs.yorku.ca-authz (Host: virt2). 2017-10-01 20:35:30,016-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmsStatisticsFetcher] (DefaultQuartzScheduler2) [3156713f] Fetched 6 VMs from VDS 'a0ccfd11-3402-4eba-a119-4987561b738e' 2017-10-01 20:35:30,017-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler2) [3156713f] VM '1734fed5-fc97-4714-bb63-55e8f4eb6d34'(wiki) moved from 'WaitForLaunch' --> 'PoweringUp' 2017-10-01 20:35:37,529-04 INFO [org.ovirt.engine.core.bll.SetVmTicketCommand] (default task-69) [819113d] Running command: SetVmTicketCommand internal: false. Entities affected : ID: 1734fed5-fc97-4714-bb63-55e8f4eb6d34 Type: VMAction group CONNECT_TO_VM with role type USER 2017-10-01 20:35:37,558-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (default task-69) [819113d] START, SetVmTicketVDSCommand(HostName = virt2, SetVmTicketVDSCommandParameters:{runAsync='true', hostId='a0ccfd11-3402-4eba-a119-4987561b738e', vmId='1734fed5-fc97-4714-bb63-55e8f4eb6d34', protocol='SPICE', ticket='Z12YJ5fcLRK6', validTime='120', userName='jas', userId='4a225b32-bc9d-42b7-90f8-f7848db0e59f', disconnectAction='LOCK_SCREEN'}), log id: 9a1af20 2017-10-01 20:35:38,606-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (default task-69) [819113d] FINISH, SetVmTicketVDSCommand, log id: 9a1af20 2017-10-01 20:35:38,614-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-69) [819113d] EVENT_ID: VM_SET_TICKET(164), Correlation ID: 819113d, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: User jas@eecs.yorku.ca-authz initiated console session for VM wiki 2017-10-01 20:35:45,914-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [3d108a21] EVENT_ID: VM_CONSOLE_CONNECTED(167), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: User jas@eecs.yorku.ca-authz is connected to VM wiki. 2017-10-01 20:36:02,132-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [3b5c2810] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 20:36:32,757-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler8) [6e8b3dc2] VM '1734fed5-fc97-4714-bb63-55e8f4eb6d34'(wiki) moved from 'PoweringUp' --> 'Up' 2017-10-01 20:36:32,768-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [6e8b3dc2] EVENT_ID: USER_RUN_VM(32), Correlation ID: 30fa922c-63b2-4b67-b376-10029722eb4a, Job ID: ba54f4e9-1e79-4f20-9967-00163a5a958a, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM wiki started on Host virt2 2017-10-01 20:37:02,152-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [740f00fa] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 20:38:02,178-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler7) [1810fb52] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 20:39:02,197-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [2ea6efdf] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 20:40:02,217-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [4ea96616] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 20:41:02,238-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler7) [135cb793] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 20:42:02,257-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [58b8d8d3] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 20:43:02,281-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [4a293077] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 20:44:02,299-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [394fa2f5] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 20:45:02,316-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [7e508b8b] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 20:45:27,141-04 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [5f7bcc5a] EVENT_ID: HOST_INTERFACE_HIGH_NETWORK_USE(608), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Host virt1 has network interface which exceeded the defined threshold [95%] (p3p1: transmit rate[0%], receive rate [97%]) 2017-10-01 20:46:02,335-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [7dfb348f] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 20:47:02,354-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [4bbda085] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 20:48:02,378-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [2a824af6] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 20:49:02,400-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [5a480b1] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 20:50:02,420-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [16a99ad7] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 20:51:02,441-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [51ec7793] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 20:52:02,461-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [4a16d465] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 20:53:02,485-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [36809703] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 20:54:02,505-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [2611d20e] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 20:55:02,525-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [13289f37] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 20:56:02,546-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [1b669377] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 20:56:27,249-04 INFO [org.ovirt.engine.core.bll.ShutdownVmCommand] (org.ovirt.thread.pool-6-thread-42) [030a73a8-68d2-4863-97cf-57c972bee0c5] Running command: ShutdownVmCommand internal: false. Entities affected : ID: d66235b4-7c8d-4da2-a572-82ecc9c53a59 Type: VMAction group SHUT_DOWN_VM with role type USER 2017-10-01 20:56:27,265-04 INFO [org.ovirt.engine.core.bll.ShutdownVmCommand] (org.ovirt.thread.pool-6-thread-42) [030a73a8-68d2-4863-97cf-57c972bee0c5] Entered (VM 'www-las'). 2017-10-01 20:56:27,265-04 INFO [org.ovirt.engine.core.bll.ShutdownVmCommand] (org.ovirt.thread.pool-6-thread-42) [030a73a8-68d2-4863-97cf-57c972bee0c5] Sending shutdown command for VM 'www-las'. 2017-10-01 20:56:27,265-04 INFO [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (org.ovirt.thread.pool-6-thread-42) [030a73a8-68d2-4863-97cf-57c972bee0c5] START, DestroyVmVDSCommand( DestroyVmVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vmId='d66235b4-7c8d-4da2-a572-82ecc9c53a59', force='false', secondsToWait='30', gracefully='true', reason='', ignoreNoVm='false'}), log id: 64ffead2 2017-10-01 20:56:27,265-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-6-thread-42) [030a73a8-68d2-4863-97cf-57c972bee0c5] START, DestroyVDSCommand(HostName = virt3, DestroyVmVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vmId='d66235b4-7c8d-4da2-a572-82ecc9c53a59', force='false', secondsToWait='30', gracefully='true', reason='', ignoreNoVm='false'}), log id: 4e72357 2017-10-01 20:56:28,269-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-6-thread-42) [030a73a8-68d2-4863-97cf-57c972bee0c5] FINISH, DestroyVDSCommand, log id: 4e72357 2017-10-01 20:56:28,272-04 INFO [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (org.ovirt.thread.pool-6-thread-42) [030a73a8-68d2-4863-97cf-57c972bee0c5] FINISH, DestroyVmVDSCommand, return: PoweringDown, log id: 64ffead2 2017-10-01 20:56:28,283-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-42) [030a73a8-68d2-4863-97cf-57c972bee0c5] EVENT_ID: USER_INITIATED_SHUTDOWN_VM(73), Correlation ID: 030a73a8-68d2-4863-97cf-57c972bee0c5, Job ID: fd79c8ab-e993-4def-9656-b94e9c0fec59, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM shutdown initiated by jas@eecs.yorku.ca-authz on VM www-las (Host: virt3). 2017-10-01 20:56:48,362-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler4) [37f99734] VM 'd66235b4-7c8d-4da2-a572-82ecc9c53a59' was reported as Down on VDS 'e1e71e9d-a0cd-47db-96c6-f724f6cea736'(virt3) 2017-10-01 20:56:48,362-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (DefaultQuartzScheduler4) [37f99734] START, DestroyVDSCommand(HostName = virt3, DestroyVmVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vmId='d66235b4-7c8d-4da2-a572-82ecc9c53a59', force='false', secondsToWait='0', gracefully='false', reason='', ignoreNoVm='true'}), log id: 7d18ab67 2017-10-01 20:56:49,367-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (DefaultQuartzScheduler4) [37f99734] FINISH, DestroyVDSCommand, log id: 7d18ab67 2017-10-01 20:56:49,367-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler4) [37f99734] VM 'd66235b4-7c8d-4da2-a572-82ecc9c53a59'(www-las) moved from 'PoweringDown' --> 'Down' 2017-10-01 20:56:49,395-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [37f99734] EVENT_ID: VM_DOWN(61), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM www-las is down. Exit message: Admin shut down from the engine 2017-10-01 20:56:49,409-04 INFO [org.ovirt.engine.core.bll.ProcessDownVmCommand] (org.ovirt.thread.pool-6-thread-16) [4611711] Running command: ProcessDownVmCommand internal: true. 2017-10-01 20:57:02,569-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler7) [36468db] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 20:57:04,415-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmsStatisticsFetcher] (DefaultQuartzScheduler7) [14e12d6c] Fetched 9 VMs from VDS 'e1e71e9d-a0cd-47db-96c6-f724f6cea736' 2017-10-01 20:58:02,594-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [7f534c43] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 20:59:02,614-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [2f0a747d] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 20:59:56,118-04 INFO [org.ovirt.engine.core.bll.RunVmCommand] (default task-192) [9b66ca7d-a14d-4ebc-ba6e-b13a4de0453b] Lock Acquired to object 'EngineLock:{exclusiveLocks='[d66235b4-7c8d-4da2-a572-82ecc9c53a59=VM]', sharedLocks=''}' 2017-10-01 20:59:56,134-04 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (default task-192) [9b66ca7d-a14d-4ebc-ba6e-b13a4de0453b] START, IsVmDuringInitiatingVDSCommand( IsVmDuringInitiatingVDSCommandParameters:{runAsync='true', vmId='d66235b4-7c8d-4da2-a572-82ecc9c53a59'}), log id: 2f9a9f40 2017-10-01 20:59:56,134-04 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (default task-192) [9b66ca7d-a14d-4ebc-ba6e-b13a4de0453b] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 2f9a9f40 2017-10-01 20:59:56,225-04 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-6-thread-19) [9b66ca7d-a14d-4ebc-ba6e-b13a4de0453b] Running command: RunVmCommand internal: false. Entities affected : ID: d66235b4-7c8d-4da2-a572-82ecc9c53a59 Type: VMAction group RUN_VM with role type USER 2017-10-01 20:59:56,309-04 INFO [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (org.ovirt.thread.pool-6-thread-19) [9b66ca7d-a14d-4ebc-ba6e-b13a4de0453b] START, UpdateVmDynamicDataVDSCommand( UpdateVmDynamicDataVDSCommandParameters:{runAsync='true', hostId='null', vmId='d66235b4-7c8d-4da2-a572-82ecc9c53a59', vmDynamic='org.ovirt.engine.core.common.businessentities.VmDynamic@96eb3023'}), log id: 57e07f4e 2017-10-01 20:59:56,311-04 INFO [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (org.ovirt.thread.pool-6-thread-19) [9b66ca7d-a14d-4ebc-ba6e-b13a4de0453b] FINISH, UpdateVmDynamicDataVDSCommand, log id: 57e07f4e 2017-10-01 20:59:56,313-04 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-6-thread-19) [9b66ca7d-a14d-4ebc-ba6e-b13a4de0453b] START, CreateVmVDSCommand( CreateVmVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vmId='d66235b4-7c8d-4da2-a572-82ecc9c53a59', vm='VM [www-las]'}), log id: 2c9201be 2017-10-01 20:59:56,314-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-6-thread-19) [9b66ca7d-a14d-4ebc-ba6e-b13a4de0453b] START, CreateVDSCommand(HostName = virt3, CreateVmVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vmId='d66235b4-7c8d-4da2-a572-82ecc9c53a59', vm='VM [www-las]'}), log id: 2878716b 2017-10-01 20:59:56,317-04 INFO [org.ovirt.engine.core.vdsbroker.builder.vminfo.VmInfoBuilderImpl] (org.ovirt.thread.pool-6-thread-19) [9b66ca7d-a14d-4ebc-ba6e-b13a4de0453b] Bootable disk '1ddc83e8-35d8-42d0-882a-d8bc35c2efa2' set to index '0' 2017-10-01 20:59:56,330-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-6-thread-19) [9b66ca7d-a14d-4ebc-ba6e-b13a4de0453b] VM {memGuaranteedSize=4096, smpThreadsPerCore=1, cpuType=SandyBridge, vmId=d66235b4-7c8d-4da2-a572-82ecc9c53a59, acpiEnable=true, vmType=kvm, smp=4, smpCoresPerSocket=1, emulatedMachine=pc-i440fx-rhel7.3.0, smartcardEnable=false, guestNumaNodes=[{memory=4096, cpus=0,1,2,3, nodeIndex=0}], transparentHugePages=true, displayNetwork=yorku, vmName=www-las, maxVCpus=16, kvmEnable=true, devices=[{address={bus=0x00, domain=0x0000, function=0x0, slot=0x02, type=pci}, type=video, specParams={heads=1, vram=32768}, device=qxl, deviceId=807d13fa-945a-492f-9bd9-00dda83650b3}, {type=graphics, specParams={fileTransferEnable=true, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,ssmartcard,susbredir, copyPasteEnable=true}, device=spice, deviceId=00000032-0032-0032-0032-0000000002c8}, {iface=ide, shared=false, path=, address={bus=1, controller=0, unit=0, type=drive, target=0}, readonly=true, index=2, type=disk, specParams={path=}, device=cdrom, deviceId=33931b19-35b1-4848-91d7-177e45ac7bee}, {discard=false, shared=false, address={bus=0x00, domain=0x0000, function=0x0, slot=0x07, type=pci}, imageID=1ddc83e8-35d8-42d0-882a-d8bc35c2efa2, format=raw, index=0, optional=false, type=disk, deviceId=1ddc83e8-35d8-42d0-882a-d8bc35c2efa2, domainID=c8653dba-63ec-4447-872c-0ffcc6af9c0f, propagateErrors=off, iface=virtio, readonly=false, bootOrder=1, poolID=aca61eac-ed8a-484b-bbee-3645148f6405, volumeID=a59c8ccc-6535-4720-80ff-92aca6b54be0, specParams={}, device=disk}, {discard=false, shared=false, address={bus=0x00, domain=0x0000, function=0x0, slot=0x08, type=pci}, imageID=aa973707-8311-4111-8f5b-1de610736bec, format=raw, optional=false, type=disk, deviceId=aa973707-8311-4111-8f5b-1de610736bec, domainID=c8653dba-63ec-4447-872c-0ffcc6af9c0f, propagateErrors=off, iface=virtio, readonly=false, poolID=aca61eac-ed8a-484b-bbee-3645148f6405, volumeID=9a634692-1302-471f-a92e-c978b2b67fd0, specParams={}, device=disk}, {filter=vdsm-no-mac-spoofing, nicModel=pv, address={bus=0x00, domain=0x0000, function=0x0, slot=0x03, type=pci}, type=interface, specParams={inbound={}, outbound={}}, device=bridge, linkActive=true, deviceId=e413fb88-ed8b-41b2-b0d0-805dd9cc38ea, macAddr=00:1a:4a:32:3d:15, network=yorku}, {filter=vdsm-no-mac-spoofing, nicModel=pv, address={bus=0x00, domain=0x0000, function=0x0, slot=0x04, type=pci}, type=interface, specParams={inbound={}, outbound={}}, device=bridge, linkActive=true, deviceId=0c13d576-ca04-42cd-bf7a-8c72af3099ae, macAddr=00:1a:4a:32:3d:16, network=mrpriv}, {address={bus=0x00, domain=0x0000, function=0x2, slot=0x01, type=pci}, index=0, model=piix3-uhci, type=controller, specParams={}, device=usb, deviceId=75dba8d3-3783-47b6-95ea-7587c028d9d9}, {address={bus=0x00, domain=0x0000, function=0x0, slot=0x05, type=pci}, index=0, model=virtio-scsi, type=controller, specParams={}, device=scsi, deviceId=879f410a-f62e-44b9-8e34-8666e5aaf087}, {address={bus=0x00, domain=0x0000, function=0x0, slot=0x06, type=pci}, type=controller, specParams={}, device=virtio-serial, deviceId=f85a1fef-5b32-46ab-aafc-194bd2a2df30}, {model=virtio, type=rng, specParams={source=urandom}, device=virtio, deviceId=4440cf64-af9a-4390-a6eb-504a8c03a12a}], custom={device_25f8c5be-f7df-4f56-9369-1ac1002995fe=VmDevice:{id='VmDeviceId:{deviceId='25f8c5be-f7df-4f56-9369-1ac1002995fe', vmId='d66235b4-7c8d-4da2-a572-82ecc9c53a59'}', device='ide', type='CONTROLLER', bootOrder='0', specParams='[]', address='{slot=0x01, bus=0x00, domain=0x0000, type=pci, function=0x1}', managed='false', plugged='true', readOnly='false', deviceAlias='ide', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_25f8c5be-f7df-4f56-9369-1ac1002995fedevice_ddbda65b-320b-4be5-b1f6-5a33508c5d1adevice_ac33b921-45d0-48ca-9844-a1380fcc3116device_b8689025-3453-4aaa-a6bb-5e6afb7d1f7d=VmDevice:{id='VmDeviceId:{deviceId='b8689025-3453-4aaa-a6bb-5e6afb7d1f7d', vmId='d66235b4-7c8d-4da2-a572-82ecc9c53a59'}', device='spicevmc', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=3}', managed='false', plugged='true', readOnly='false', deviceAlias='channel2', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_25f8c5be-f7df-4f56-9369-1ac1002995fedevice_ddbda65b-320b-4be5-b1f6-5a33508c5d1adevice_ac33b921-45d0-48ca-9844-a1380fcc3116=VmDevice:{id='VmDeviceId:{deviceId='ac33b921-45d0-48ca-9844-a1380fcc3116', vmId='d66235b4-7c8d-4da2-a572-82ecc9c53a59'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=2}', managed='false', plugged='true', readOnly='false', deviceAlias='channel1', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_25f8c5be-f7df-4f56-9369-1ac1002995fedevice_ddbda65b-320b-4be5-b1f6-5a33508c5d1a=VmDevice:{id='VmDeviceId:{deviceId='ddbda65b-320b-4be5-b1f6-5a33508c5d1a', vmId='d66235b4-7c8d-4da2-a572-82ecc9c53a59'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=1}', managed='false', plugged='true', readOnly='false', deviceAlias='channel0', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}}, display=qxl, timeOffset=0, nice=0, maxMemSize=16384, maxMemSlots=16, bootMenuEnable=false, memSize=4096} 2017-10-01 20:59:57,084-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-6-thread-19) [9b66ca7d-a14d-4ebc-ba6e-b13a4de0453b] FINISH, CreateVDSCommand, log id: 2878716b 2017-10-01 20:59:57,088-04 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-6-thread-19) [9b66ca7d-a14d-4ebc-ba6e-b13a4de0453b] FINISH, CreateVmVDSCommand, return: WaitForLaunch, log id: 2c9201be 2017-10-01 20:59:57,089-04 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-6-thread-19) [9b66ca7d-a14d-4ebc-ba6e-b13a4de0453b] Lock freed to object 'EngineLock:{exclusiveLocks='[d66235b4-7c8d-4da2-a572-82ecc9c53a59=VM]', sharedLocks=''}' 2017-10-01 20:59:57,098-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-19) [9b66ca7d-a14d-4ebc-ba6e-b13a4de0453b] EVENT_ID: USER_STARTED_VM(153), Correlation ID: 9b66ca7d-a14d-4ebc-ba6e-b13a4de0453b, Job ID: 7685df13-e441-45b6-a763-c549bdf8e6da, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM www-las was started by jas@eecs.yorku.ca-authz (Host: virt3). 2017-10-01 21:00:02,631-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [1320b758] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 21:00:04,864-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmsStatisticsFetcher] (DefaultQuartzScheduler10) [4b74505b] Fetched 10 VMs from VDS 'e1e71e9d-a0cd-47db-96c6-f724f6cea736' 2017-10-01 21:00:04,866-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler10) [4b74505b] VM 'd66235b4-7c8d-4da2-a572-82ecc9c53a59'(www-las) moved from 'WaitForLaunch' --> 'PoweringUp' 2017-10-01 21:00:22,491-04 INFO [org.ovirt.engine.core.bll.SetVmTicketCommand] (default task-223) [73928c3b] Running command: SetVmTicketCommand internal: false. Entities affected : ID: d66235b4-7c8d-4da2-a572-82ecc9c53a59 Type: VMAction group CONNECT_TO_VM with role type USER 2017-10-01 21:00:22,521-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (default task-223) [73928c3b] START, SetVmTicketVDSCommand(HostName = virt3, SetVmTicketVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vmId='d66235b4-7c8d-4da2-a572-82ecc9c53a59', protocol='SPICE', ticket='56Ex+rlmrtB1', validTime='120', userName='jas', userId='4a225b32-bc9d-42b7-90f8-f7848db0e59f', disconnectAction='LOCK_SCREEN'}), log id: 470a0ce1 2017-10-01 21:00:22,549-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (default task-223) [73928c3b] FINISH, SetVmTicketVDSCommand, log id: 470a0ce1 2017-10-01 21:00:22,558-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-223) [73928c3b] EVENT_ID: VM_SET_TICKET(164), Correlation ID: 73928c3b, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: User jas@eecs.yorku.ca-authz initiated console session for VM www-las 2017-10-01 21:00:34,970-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [5d87d65d] EVENT_ID: VM_CONSOLE_CONNECTED(167), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: User jas@eecs.yorku.ca-authz is connected to VM www-las. 2017-10-01 21:00:59,983-04 INFO [org.ovirt.engine.core.bll.ShutdownVmCommand] (org.ovirt.thread.pool-6-thread-50) [ebf96992-1729-43ca-a8dc-0d3c340a2f61] Running command: ShutdownVmCommand internal: false. Entities affected : ID: d56363ad-9cb1-4a70-a33a-ce439b044d7d Type: VMAction group SHUT_DOWN_VM with role type USER 2017-10-01 21:00:59,998-04 INFO [org.ovirt.engine.core.bll.ShutdownVmCommand] (org.ovirt.thread.pool-6-thread-50) [ebf96992-1729-43ca-a8dc-0d3c340a2f61] Entered (VM 'svn'). 2017-10-01 21:00:59,999-04 INFO [org.ovirt.engine.core.bll.ShutdownVmCommand] (org.ovirt.thread.pool-6-thread-50) [ebf96992-1729-43ca-a8dc-0d3c340a2f61] Sending shutdown command for VM 'svn'. 2017-10-01 21:00:59,999-04 INFO [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ebf96992-1729-43ca-a8dc-0d3c340a2f61] START, DestroyVmVDSCommand( DestroyVmVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vmId='d56363ad-9cb1-4a70-a33a-ce439b044d7d', force='false', secondsToWait='30', gracefully='true', reason='', ignoreNoVm='false'}), log id: 349c34e1 2017-10-01 21:00:59,999-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ebf96992-1729-43ca-a8dc-0d3c340a2f61] START, DestroyVDSCommand(HostName = virt3, DestroyVmVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vmId='d56363ad-9cb1-4a70-a33a-ce439b044d7d', force='false', secondsToWait='30', gracefully='true', reason='', ignoreNoVm='false'}), log id: 31489b3 2017-10-01 21:01:01,003-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ebf96992-1729-43ca-a8dc-0d3c340a2f61] FINISH, DestroyVDSCommand, log id: 31489b3 2017-10-01 21:01:01,005-04 INFO [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (org.ovirt.thread.pool-6-thread-50) [ebf96992-1729-43ca-a8dc-0d3c340a2f61] FINISH, DestroyVmVDSCommand, return: PoweringDown, log id: 349c34e1 2017-10-01 21:01:01,010-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-50) [ebf96992-1729-43ca-a8dc-0d3c340a2f61] EVENT_ID: USER_INITIATED_SHUTDOWN_VM(73), Correlation ID: ebf96992-1729-43ca-a8dc-0d3c340a2f61, Job ID: ee45c621-6bdb-48a4-a2d9-1c89ed3c9a44, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM shutdown initiated by jas@eecs.yorku.ca-authz on VM svn (Host: virt3). 2017-10-01 21:01:02,647-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [6f655509] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 21:01:05,042-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [3a6c9032] EVENT_ID: VM_CONSOLE_DISCONNECTED(168), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: User jas@eecs.yorku.ca-authz got disconnected from VM www-las. 2017-10-01 21:01:05,042-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler10) [3a6c9032] VM 'd66235b4-7c8d-4da2-a572-82ecc9c53a59'(www-las) moved from 'PoweringUp' --> 'Up' 2017-10-01 21:01:05,054-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler10) [3a6c9032] EVENT_ID: USER_RUN_VM(32), Correlation ID: 9b66ca7d-a14d-4ebc-ba6e-b13a4de0453b, Job ID: 7685df13-e441-45b6-a763-c549bdf8e6da, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM www-las started on Host virt3 2017-10-01 21:01:20,072-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler6) [70503b7b] VM 'd56363ad-9cb1-4a70-a33a-ce439b044d7d' was reported as Down on VDS 'e1e71e9d-a0cd-47db-96c6-f724f6cea736'(virt3) 2017-10-01 21:01:20,072-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (DefaultQuartzScheduler6) [70503b7b] START, DestroyVDSCommand(HostName = virt3, DestroyVmVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vmId='d56363ad-9cb1-4a70-a33a-ce439b044d7d', force='false', secondsToWait='0', gracefully='false', reason='', ignoreNoVm='true'}), log id: 2f04e205 2017-10-01 21:01:21,076-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (DefaultQuartzScheduler6) [70503b7b] FINISH, DestroyVDSCommand, log id: 2f04e205 2017-10-01 21:01:21,076-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler6) [70503b7b] VM 'd56363ad-9cb1-4a70-a33a-ce439b044d7d'(svn) moved from 'PoweringDown' --> 'Down' 2017-10-01 21:01:21,098-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [70503b7b] EVENT_ID: VM_DOWN(61), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM svn is down. Exit message: Admin shut down from the engine 2017-10-01 21:01:21,111-04 INFO [org.ovirt.engine.core.bll.ProcessDownVmCommand] (org.ovirt.thread.pool-6-thread-45) [9e9805d] Running command: ProcessDownVmCommand internal: true. 2017-10-01 21:01:36,117-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmsStatisticsFetcher] (DefaultQuartzScheduler3) [68fffae] Fetched 9 VMs from VDS 'e1e71e9d-a0cd-47db-96c6-f724f6cea736' 2017-10-01 21:02:02,669-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [3eaccb6] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 21:03:02,696-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler7) [3667cc90] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 21:03:14,265-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler2) [4ba00b9a] EVENT_ID: VM_CONSOLE_DISCONNECTED(168), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: User jas@eecs.yorku.ca-authz got disconnected from VM wiki. 2017-10-01 21:04:02,716-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [6eb1ddb] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 21:05:02,736-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [13c22dc6] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 21:06:02,756-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [5823d611] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 21:07:02,775-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [40888aab] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 21:08:02,803-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [6bf68270] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 21:09:02,820-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [3846fae3] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 21:10:02,839-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [5fa8e9e9] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 21:11:02,859-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [290b55b7] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 21:12:02,878-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [429035d9] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 21:13:02,903-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [4d4d09c2] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 21:14:02,920-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [4ed6d307] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 21:15:02,937-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [29442b17] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 21:16:02,960-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler7) [15095e39] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 21:16:22,266-04 INFO [org.ovirt.engine.core.bll.RunVmCommand] (default task-313) [498b7e92-518c-44cf-8a60-488c63cc29d4] Lock Acquired to object 'EngineLock:{exclusiveLocks='[d56363ad-9cb1-4a70-a33a-ce439b044d7d=VM]', sharedLocks=''}' 2017-10-01 21:16:22,291-04 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (default task-313) [498b7e92-518c-44cf-8a60-488c63cc29d4] START, IsVmDuringInitiatingVDSCommand( IsVmDuringInitiatingVDSCommandParameters:{runAsync='true', vmId='d56363ad-9cb1-4a70-a33a-ce439b044d7d'}), log id: d4970a4 2017-10-01 21:16:22,292-04 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (default task-313) [498b7e92-518c-44cf-8a60-488c63cc29d4] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: d4970a4 2017-10-01 21:16:22,385-04 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-6-thread-21) [498b7e92-518c-44cf-8a60-488c63cc29d4] Running command: RunVmCommand internal: false. Entities affected : ID: d56363ad-9cb1-4a70-a33a-ce439b044d7d Type: VMAction group RUN_VM with role type USER 2017-10-01 21:16:22,481-04 INFO [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (org.ovirt.thread.pool-6-thread-21) [498b7e92-518c-44cf-8a60-488c63cc29d4] START, UpdateVmDynamicDataVDSCommand( UpdateVmDynamicDataVDSCommandParameters:{runAsync='true', hostId='null', vmId='d56363ad-9cb1-4a70-a33a-ce439b044d7d', vmDynamic='org.ovirt.engine.core.common.businessentities.VmDynamic@db9f9f22'}), log id: 70d06d82 2017-10-01 21:16:22,493-04 INFO [org.ovirt.engine.core.vdsbroker.UpdateVmDynamicDataVDSCommand] (org.ovirt.thread.pool-6-thread-21) [498b7e92-518c-44cf-8a60-488c63cc29d4] FINISH, UpdateVmDynamicDataVDSCommand, log id: 70d06d82 2017-10-01 21:16:22,496-04 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-6-thread-21) [498b7e92-518c-44cf-8a60-488c63cc29d4] START, CreateVmVDSCommand( CreateVmVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vmId='d56363ad-9cb1-4a70-a33a-ce439b044d7d', vm='VM [svn]'}), log id: 77c3b818 2017-10-01 21:16:22,499-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-6-thread-21) [498b7e92-518c-44cf-8a60-488c63cc29d4] START, CreateVDSCommand(HostName = virt3, CreateVmVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vmId='d56363ad-9cb1-4a70-a33a-ce439b044d7d', vm='VM [svn]'}), log id: 1255ade0 2017-10-01 21:16:22,503-04 INFO [org.ovirt.engine.core.vdsbroker.builder.vminfo.VmInfoBuilderImpl] (org.ovirt.thread.pool-6-thread-21) [498b7e92-518c-44cf-8a60-488c63cc29d4] Bootable disk 'fcae8e52-ab63-4d21-8018-8a90ebb1f35a' set to index '0' 2017-10-01 21:16:22,523-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-6-thread-21) [498b7e92-518c-44cf-8a60-488c63cc29d4] VM {memGuaranteedSize=4096, smpThreadsPerCore=1, cpuType=SandyBridge, vmId=d56363ad-9cb1-4a70-a33a-ce439b044d7d, acpiEnable=true, vmType=kvm, smp=4, smpCoresPerSocket=1, emulatedMachine=pc-i440fx-rhel7.3.0, smartcardEnable=false, guestNumaNodes=[{memory=4096, cpus=0,1,2,3, nodeIndex=0}], transparentHugePages=true, displayNetwork=yorku, vmName=svn, maxVCpus=16, kvmEnable=true, devices=[{address={bus=0x00, domain=0x0000, function=0x0, slot=0x02, type=pci}, type=video, specParams={heads=1, vram=32768}, device=qxl, deviceId=17a773ff-d339-4628-9e8d-8bfe62550932}, {type=graphics, specParams={fileTransferEnable=true, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,ssmartcard,susbredir, copyPasteEnable=true}, device=spice, deviceId=0000002e-002e-002e-002e-00000000036d}, {iface=ide, shared=false, path=, address={bus=1, controller=0, unit=0, type=drive, target=0}, readonly=true, index=2, type=disk, specParams={path=}, device=cdrom, deviceId=90de1795-9555-479b-8a82-5bcc7efa17d3}, {discard=false, shared=false, address={bus=0x00, domain=0x0000, function=0x0, slot=0x07, type=pci}, imageID=fcae8e52-ab63-4d21-8018-8a90ebb1f35a, format=raw, index=0, optional=false, type=disk, deviceId=fcae8e52-ab63-4d21-8018-8a90ebb1f35a, domainID=c8653dba-63ec-4447-872c-0ffcc6af9c0f, propagateErrors=off, iface=virtio, readonly=false, bootOrder=1, poolID=aca61eac-ed8a-484b-bbee-3645148f6405, volumeID=d6357103-21d8-4547-a270-9f411cbc84bc, specParams={}, device=disk}, {discard=false, shared=false, address={bus=0x00, domain=0x0000, function=0x0, slot=0x08, type=pci}, imageID=bebf0874-fcc4-4acf-84da-0d6898a3900a, format=raw, optional=false, type=disk, deviceId=bebf0874-fcc4-4acf-84da-0d6898a3900a, domainID=c8653dba-63ec-4447-872c-0ffcc6af9c0f, propagateErrors=off, iface=virtio, readonly=false, poolID=aca61eac-ed8a-484b-bbee-3645148f6405, volumeID=8b9860ba-be20-407a-a272-a0bf82bde445, specParams={}, device=disk}, {filter=vdsm-no-mac-spoofing, nicModel=pv, address={bus=0x00, domain=0x0000, function=0x0, slot=0x03, type=pci}, type=interface, specParams={inbound={}, outbound={}}, device=bridge, linkActive=true, deviceId=6a766172-f4b1-4ed4-84ee-b74b65ff8d4c, macAddr=00:1a:4a:32:3d:11, network=yorku}, {filter=vdsm-no-mac-spoofing, nicModel=pv, address={bus=0x00, domain=0x0000, function=0x0, slot=0x04, type=pci}, type=interface, specParams={inbound={}, outbound={}}, device=bridge, linkActive=true, deviceId=117cf962-4ff9-4b09-9383-741ee1f43b54, macAddr=00:1a:4a:32:3d:12, network=mrpriv}, {address={bus=0x00, domain=0x0000, function=0x2, slot=0x01, type=pci}, index=0, model=piix3-uhci, type=controller, specParams={}, device=usb, deviceId=23cbb675-029c-4772-bc9f-a315edd41b21}, {address={bus=0x00, domain=0x0000, function=0x0, slot=0x05, type=pci}, index=0, model=virtio-scsi, type=controller, specParams={}, device=scsi, deviceId=8e4c754b-8ac6-4584-8fe1-4dd08406accb}, {address={bus=0x00, domain=0x0000, function=0x0, slot=0x06, type=pci}, type=controller, specParams={}, device=virtio-serial, deviceId=0f3830bc-9371-4620-bd33-d7170b74e33d}, {model=virtio, type=rng, specParams={source=urandom}, device=virtio, deviceId=0cb162a4-17c7-4e63-8ad2-00c28056a797}], custom={device_cfafea07-e88d-4aff-886a-b9ac801122dddevice_ad6f9631-fe5a-40ca-be30-b1c8caeb03e1device_5832746b-6121-4199-b998-608687567ce6=VmDevice:{id='VmDeviceId:{deviceId='5832746b-6121-4199-b998-608687567ce6', vmId='d56363ad-9cb1-4a70-a33a-ce439b044d7d'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=2}', managed='false', plugged='true', readOnly='false', deviceAlias='channel1', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_cfafea07-e88d-4aff-886a-b9ac801122dd=VmDevice:{id='VmDeviceId:{deviceId='cfafea07-e88d-4aff-886a-b9ac801122dd', vmId='d56363ad-9cb1-4a70-a33a-ce439b044d7d'}', device='ide', type='CONTROLLER', bootOrder='0', specParams='[]', address='{slot=0x01, bus=0x00, domain=0x0000, type=pci, function=0x1}', managed='false', plugged='true', readOnly='false', deviceAlias='ide', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_cfafea07-e88d-4aff-886a-b9ac801122dddevice_ad6f9631-fe5a-40ca-be30-b1c8caeb03e1device_5832746b-6121-4199-b998-608687567ce6device_9faaa84a-3bed-44b1-a853-477b55333338=VmDevice:{id='VmDeviceId:{deviceId='9faaa84a-3bed-44b1-a853-477b55333338', vmId='d56363ad-9cb1-4a70-a33a-ce439b044d7d'}', device='spicevmc', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=3}', managed='false', plugged='true', readOnly='false', deviceAlias='channel2', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_cfafea07-e88d-4aff-886a-b9ac801122dddevice_ad6f9631-fe5a-40ca-be30-b1c8caeb03e1=VmDevice:{id='VmDeviceId:{deviceId='ad6f9631-fe5a-40ca-be30-b1c8caeb03e1', vmId='d56363ad-9cb1-4a70-a33a-ce439b044d7d'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=1}', managed='false', plugged='true', readOnly='false', deviceAlias='channel0', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}}, display=qxl, timeOffset=0, nice=0, maxMemSize=16384, maxMemSlots=16, bootMenuEnable=false, memSize=4096} 2017-10-01 21:16:22,687-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (org.ovirt.thread.pool-6-thread-21) [498b7e92-518c-44cf-8a60-488c63cc29d4] FINISH, CreateVDSCommand, log id: 1255ade0 2017-10-01 21:16:22,690-04 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (org.ovirt.thread.pool-6-thread-21) [498b7e92-518c-44cf-8a60-488c63cc29d4] FINISH, CreateVmVDSCommand, return: WaitForLaunch, log id: 77c3b818 2017-10-01 21:16:22,691-04 INFO [org.ovirt.engine.core.bll.RunVmCommand] (org.ovirt.thread.pool-6-thread-21) [498b7e92-518c-44cf-8a60-488c63cc29d4] Lock freed to object 'EngineLock:{exclusiveLocks='[d56363ad-9cb1-4a70-a33a-ce439b044d7d=VM]', sharedLocks=''}' 2017-10-01 21:16:22,702-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-21) [498b7e92-518c-44cf-8a60-488c63cc29d4] EVENT_ID: USER_STARTED_VM(153), Correlation ID: 498b7e92-518c-44cf-8a60-488c63cc29d4, Job ID: 1d47b75b-332b-4f11-87a0-fbce55e67ecd, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM svn was started by jas@eecs.yorku.ca-authz (Host: virt3). 2017-10-01 21:16:22,725-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmsStatisticsFetcher] (DefaultQuartzScheduler2) [2421b7f5] Fetched 10 VMs from VDS 'e1e71e9d-a0cd-47db-96c6-f724f6cea736' 2017-10-01 21:16:22,741-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVDSCommand] (DefaultQuartzScheduler2) [2421b7f5] START, FullListVDSCommand(HostName = virt3, FullListVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vmIds='[d56363ad-9cb1-4a70-a33a-ce439b044d7d]'}), log id: 5c3380fa 2017-10-01 21:16:23,263-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVDSCommand] (DefaultQuartzScheduler2) [2421b7f5] FINISH, FullListVDSCommand, return: [{acpiEnable=true, emulatedMachine=pc-i440fx-rhel7.3.0, vmId=d56363ad-9cb1-4a70-a33a-ce439b044d7d, guestDiskMapping={}, transparentHugePages=true, timeOffset=0, cpuType=SandyBridge, smp=4, guestNumaNodes=[Ljava.lang.Object;@12e9e8d3, smartcardEnable=false, custom={device_cfafea07-e88d-4aff-886a-b9ac801122dddevice_ad6f9631-fe5a-40ca-be30-b1c8caeb03e1device_5832746b-6121-4199-b998-608687567ce6=VmDevice:{id='VmDeviceId:{deviceId='5832746b-6121-4199-b998-608687567ce6', vmId='d56363ad-9cb1-4a70-a33a-ce439b044d7d'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=2}', managed='false', plugged='true', readOnly='false', deviceAlias='channel1', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_cfafea07-e88d-4aff-886a-b9ac801122dd=VmDevice:{id='VmDeviceId:{deviceId='cfafea07-e88d-4aff-886a-b9ac801122dd', vmId='d56363ad-9cb1-4a70-a33a-ce439b044d7d'}', device='ide', type='CONTROLLER', bootOrder='0', specParams='[]', address='{slot=0x01, bus=0x00, domain=0x0000, type=pci, function=0x1}', managed='false', plugged='true', readOnly='false', deviceAlias='ide', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_cfafea07-e88d-4aff-886a-b9ac801122dddevice_ad6f9631-fe5a-40ca-be30-b1c8caeb03e1=VmDevice:{id='VmDeviceId:{deviceId='ad6f9631-fe5a-40ca-be30-b1c8caeb03e1', vmId='d56363ad-9cb1-4a70-a33a-ce439b044d7d'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=1}', managed='false', plugged='true', readOnly='false', deviceAlias='channel0', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_cfafea07-e88d-4aff-886a-b9ac801122dddevice_ad6f9631-fe5a-40ca-be30-b1c8caeb03e1device_5832746b-6121-4199-b998-608687567ce6device_9faaa84a-3bed-44b1-a853-477b55333338=VmDevice:{id='VmDeviceId:{deviceId='9faaa84a-3bed-44b1-a853-477b55333338', vmId='d56363ad-9cb1-4a70-a33a-ce439b044d7d'}', device='spicevmc', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=3}', managed='false', plugged='true', readOnly='false', deviceAlias='channel2', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}}, vmType=kvm, memSize=4096, smpCoresPerSocket=1, vmName=svn, nice=0, status=WaitForLaunch, maxMemSize=16384, bootMenuEnable=false, pid=0, smpThreadsPerCore=1, memGuaranteedSize=4096, kvmEnable=true, displayNetwork=yorku, devices=[Ljava.lang.Object;@78f9918b, display=qxl, maxVCpus=16, clientIp=, statusTime=5279599200, maxMemSlots=16}], log id: 5c3380fa 2017-10-01 21:16:23,265-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler2) [2421b7f5] Received a memballoon Device without an address when processing VM d56363ad-9cb1-4a70-a33a-ce439b044d7d devices, skipping device: {device=memballoon, specParams={model=none}, type=balloon} 2017-10-01 21:16:23,265-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler2) [2421b7f5] Received a virtio Device without an address when processing VM d56363ad-9cb1-4a70-a33a-ce439b044d7d devices, skipping device: {device=virtio, specParams={source=urandom}, model=virtio, type=rng, deviceId=0cb162a4-17c7-4e63-8ad2-00c28056a797} 2017-10-01 21:16:23,265-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler2) [2421b7f5] Received a spice Device without an address when processing VM d56363ad-9cb1-4a70-a33a-ce439b044d7d devices, skipping device: {device=spice, specParams={fileTransferEnable=true, displayNetwork=yorku, copyPasteEnable=true, displayIp=130.63.94.36, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,ssmartcard,susbredir}, type=graphics, deviceId=0000002e-002e-002e-002e-00000000036d} 2017-10-01 21:16:38,290-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler7) [32eebcc6] VM 'd56363ad-9cb1-4a70-a33a-ce439b044d7d'(svn) moved from 'WaitForLaunch' --> 'PoweringUp' 2017-10-01 21:16:38,303-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVDSCommand] (DefaultQuartzScheduler7) [32eebcc6] START, FullListVDSCommand(HostName = virt3, FullListVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vmIds='[d56363ad-9cb1-4a70-a33a-ce439b044d7d]'}), log id: 126a6ea7 2017-10-01 21:16:39,319-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVDSCommand] (DefaultQuartzScheduler7) [32eebcc6] FINISH, FullListVDSCommand, return: [{acpiEnable=true, emulatedMachine=pc-i440fx-rhel7.3.0, vmId=d56363ad-9cb1-4a70-a33a-ce439b044d7d, guestDiskMapping={}, transparentHugePages=true, timeOffset=0, cpuType=SandyBridge, smp=4, pauseCode=NOERR, guestNumaNodes=[Ljava.lang.Object;@48c6a899, smartcardEnable=false, custom={device_cfafea07-e88d-4aff-886a-b9ac801122dddevice_ad6f9631-fe5a-40ca-be30-b1c8caeb03e1device_5832746b-6121-4199-b998-608687567ce6=VmDevice:{id='VmDeviceId:{deviceId='5832746b-6121-4199-b998-608687567ce6', vmId='d56363ad-9cb1-4a70-a33a-ce439b044d7d'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=2}', managed='false', plugged='true', readOnly='false', deviceAlias='channel1', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_cfafea07-e88d-4aff-886a-b9ac801122dd=VmDevice:{id='VmDeviceId:{deviceId='cfafea07-e88d-4aff-886a-b9ac801122dd', vmId='d56363ad-9cb1-4a70-a33a-ce439b044d7d'}', device='ide', type='CONTROLLER', bootOrder='0', specParams='[]', address='{slot=0x01, bus=0x00, domain=0x0000, type=pci, function=0x1}', managed='false', plugged='true', readOnly='false', deviceAlias='ide', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_cfafea07-e88d-4aff-886a-b9ac801122dddevice_ad6f9631-fe5a-40ca-be30-b1c8caeb03e1=VmDevice:{id='VmDeviceId:{deviceId='ad6f9631-fe5a-40ca-be30-b1c8caeb03e1', vmId='d56363ad-9cb1-4a70-a33a-ce439b044d7d'}', device='unix', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=1}', managed='false', plugged='true', readOnly='false', deviceAlias='channel0', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}, device_cfafea07-e88d-4aff-886a-b9ac801122dddevice_ad6f9631-fe5a-40ca-be30-b1c8caeb03e1device_5832746b-6121-4199-b998-608687567ce6device_9faaa84a-3bed-44b1-a853-477b55333338=VmDevice:{id='VmDeviceId:{deviceId='9faaa84a-3bed-44b1-a853-477b55333338', vmId='d56363ad-9cb1-4a70-a33a-ce439b044d7d'}', device='spicevmc', type='CHANNEL', bootOrder='0', specParams='[]', address='{bus=0, controller=0, type=virtio-serial, port=3}', managed='false', plugged='true', readOnly='false', deviceAlias='channel2', customProperties='[]', snapshotId='null', logicalName='null', hostDevice='null'}}, vmType=kvm, memSize=4096, smpCoresPerSocket=1, vmName=svn, nice=0, status=Up, maxMemSize=16384, bootMenuEnable=false, pid=40555, smpThreadsPerCore=1, memGuaranteedSize=4096, kvmEnable=true, displayNetwork=yorku, devices=[Ljava.lang.Object;@29ff4d65, display=qxl, maxVCpus=16, clientIp=, statusTime=5279614760, maxMemSlots=16}], log id: 126a6ea7 2017-10-01 21:16:39,322-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler7) [32eebcc6] Received a memballoon Device without an address when processing VM d56363ad-9cb1-4a70-a33a-ce439b044d7d devices, skipping device: {device=memballoon, specParams={model=none}, type=balloon, alias=balloon0} 2017-10-01 21:16:39,322-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmDevicesMonitoring] (DefaultQuartzScheduler7) [32eebcc6] Received a spice Device without an address when processing VM d56363ad-9cb1-4a70-a33a-ce439b044d7d devices, skipping device: {device=spice, specParams={fileTransferEnable=true, displayNetwork=yorku, copyPasteEnable=true, displayIp=130.63.94.36, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,ssmartcard,susbredir}, type=graphics, deviceId=0000002e-002e-002e-002e-00000000036d, tlsPort=5902} 2017-10-01 21:16:47,636-04 INFO [org.ovirt.engine.core.bll.SetVmTicketCommand] (default task-348) [3a6ecc8c] Running command: SetVmTicketCommand internal: false. Entities affected : ID: d56363ad-9cb1-4a70-a33a-ce439b044d7d Type: VMAction group CONNECT_TO_VM with role type USER 2017-10-01 21:16:47,666-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (default task-348) [3a6ecc8c] START, SetVmTicketVDSCommand(HostName = virt3, SetVmTicketVDSCommandParameters:{runAsync='true', hostId='e1e71e9d-a0cd-47db-96c6-f724f6cea736', vmId='d56363ad-9cb1-4a70-a33a-ce439b044d7d', protocol='SPICE', ticket='X6JpsM9bylNu', validTime='120', userName='jas', userId='4a225b32-bc9d-42b7-90f8-f7848db0e59f', disconnectAction='LOCK_SCREEN'}), log id: 5aeb27d1 2017-10-01 21:16:47,694-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (default task-348) [3a6ecc8c] FINISH, SetVmTicketVDSCommand, log id: 5aeb27d1 2017-10-01 21:16:47,703-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-348) [3a6ecc8c] EVENT_ID: VM_SET_TICKET(164), Correlation ID: 3a6ecc8c, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: User jas@eecs.yorku.ca-authz initiated console session for VM svn 2017-10-01 21:16:54,370-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler8) [500d9737] EVENT_ID: VM_CONSOLE_CONNECTED(167), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: User jas@eecs.yorku.ca-authz is connected to VM svn. 2017-10-01 21:17:02,980-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [37338725] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 21:17:24,453-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [79a17e3b] EVENT_ID: VM_CONSOLE_DISCONNECTED(168), Correlation ID: null, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: User jas@eecs.yorku.ca-authz got disconnected from VM svn. 2017-10-01 21:17:24,453-04 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler6) [79a17e3b] VM 'd56363ad-9cb1-4a70-a33a-ce439b044d7d'(svn) moved from 'PoweringUp' --> 'Up' 2017-10-01 21:17:24,464-04 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler6) [79a17e3b] EVENT_ID: USER_RUN_VM(32), Correlation ID: 498b7e92-518c-44cf-8a60-488c63cc29d4, Job ID: 1d47b75b-332b-4f11-87a0-fbce55e67ecd, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: VM svn started on Host virt3 2017-10-01 21:18:03,005-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler7) [4327b211] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 21:19:03,025-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [5dfa5ec2] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 21:20:03,048-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler7) [25d82bae] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 21:20:36,803-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.OvfDataUpdater] (DefaultQuartzScheduler1) [2a9253f8] Attempting to update VMs/Templates Ovf. 2017-10-01 21:20:36,806-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler1) [30bd5a19] Before acquiring and wait lock 'EngineLock:{exclusiveLocks='[aca61eac-ed8a-484b-bbee-3645148f6405=OVF_UPDATE]', sharedLocks=''}' 2017-10-01 21:20:36,806-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler1) [30bd5a19] Lock-wait acquired to object 'EngineLock:{exclusiveLocks='[aca61eac-ed8a-484b-bbee-3645148f6405=OVF_UPDATE]', sharedLocks=''}' 2017-10-01 21:20:36,807-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler1) [30bd5a19] Running command: ProcessOvfUpdateForStoragePoolCommand internal: true. Entities affected : ID: aca61eac-ed8a-484b-bbee-3645148f6405 Type: StoragePool 2017-10-01 21:20:36,874-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler1) [30bd5a19] Attempting to update VM OVFs in Data Center 'EECS-Primary' 2017-10-01 21:20:36,946-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler1) [30bd5a19] Successfully updated VM OVFs in Data Center 'EECS-Primary' 2017-10-01 21:20:36,946-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler1) [30bd5a19] Attempting to update template OVFs in Data Center 'EECS-Primary' 2017-10-01 21:20:36,947-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler1) [30bd5a19] Successfully updated templates OVFs in Data Center 'EECS-Primary' 2017-10-01 21:20:36,947-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler1) [30bd5a19] Attempting to remove unneeded template/vm OVFs in Data Center 'EECS-Primary' 2017-10-01 21:20:36,949-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler1) [30bd5a19] Successfully removed unneeded template/vm OVFs in Data Center 'EECS-Primary' 2017-10-01 21:20:36,950-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler1) [30bd5a19] Lock freed to object 'EngineLock:{exclusiveLocks='[aca61eac-ed8a-484b-bbee-3645148f6405=OVF_UPDATE]', sharedLocks=''}' 2017-10-01 21:20:36,974-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStorageDomainCommand] (DefaultQuartzScheduler1) [37bc032f] Lock Acquired to object 'EngineLock:{exclusiveLocks='[c8653dba-63ec-4447-872c-0ffcc6af9c0f=STORAGE]', sharedLocks='[aca61eac-ed8a-484b-bbee-3645148f6405=OVF_UPDATE]'}' 2017-10-01 21:20:36,990-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStorageDomainCommand] (DefaultQuartzScheduler1) [37bc032f] Running command: ProcessOvfUpdateForStorageDomainCommand internal: true. Entities affected : ID: c8653dba-63ec-4447-872c-0ffcc6af9c0f Type: StorageAction group MANIPULATE_STORAGE_DOMAIN with role type ADMIN 2017-10-01 21:20:37,008-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler1) [37bc032f] START, SetVolumeDescriptionVDSCommand( SetVolumeDescriptionVDSCommandParameters:{runAsync='true', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', ignoreFailoverLimit='false', storageDomainId='c8653dba-63ec-4447-872c-0ffcc6af9c0f', imageGroupId='bbbcf706-d019-4cc5-8096-ddf6d9436553', imageId='6c73cd57-01c6-4541-8157-495e5182f916'}), log id: 7dff4a24 2017-10-01 21:20:37,008-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler1) [37bc032f] -- executeIrsBrokerCommand: calling 'setVolumeDescription', parameters: 2017-10-01 21:20:37,008-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler1) [37bc032f] ++ spUUID=aca61eac-ed8a-484b-bbee-3645148f6405 2017-10-01 21:20:37,008-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler1) [37bc032f] ++ sdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f 2017-10-01 21:20:37,008-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler1) [37bc032f] ++ imageGroupGUID=bbbcf706-d019-4cc5-8096-ddf6d9436553 2017-10-01 21:20:37,008-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler1) [37bc032f] ++ volUUID=6c73cd57-01c6-4541-8157-495e5182f916 2017-10-01 21:20:37,008-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler1) [37bc032f] ++ description={"Updated":false,"Last Updated":"Sun Oct 01 17:20:31 EDT 2017","Storage Domains":[{"uuid":"c8653dba-63ec-4447-872c-0ffcc6af9c0f"}],"Disk Description":"OVF_STORE"} 2017-10-01 21:20:38,035-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler1) [37bc032f] FINISH, SetVolumeDescriptionVDSCommand, log id: 7dff4a24 2017-10-01 21:20:38,044-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.UploadStreamCommand] (DefaultQuartzScheduler1) [37bc032f] Lock Acquired to object 'EngineLock:{exclusiveLocks='', sharedLocks='[45f8b331-842e-48e7-9df8-56adddb93836=VDS_EXECUTION]'}' 2017-10-01 21:20:38,060-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.UploadStreamCommand] (DefaultQuartzScheduler1) [37bc032f] Running command: UploadStreamCommand internal: true. Entities affected : ID: c8653dba-63ec-4447-872c-0ffcc6af9c0f Type: Storage 2017-10-01 21:20:38,060-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (DefaultQuartzScheduler1) [37bc032f] START, UploadStreamVDSCommand(HostName = virt1, UploadStreamVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836'}), log id: 419e99b9 2017-10-01 21:20:38,060-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (DefaultQuartzScheduler1) [37bc032f] -- executeVdsBrokerCommand, parameters: 2017-10-01 21:20:38,060-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (DefaultQuartzScheduler1) [37bc032f] ++ spUUID=aca61eac-ed8a-484b-bbee-3645148f6405 2017-10-01 21:20:38,060-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (DefaultQuartzScheduler1) [37bc032f] ++ sdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f 2017-10-01 21:20:38,060-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (DefaultQuartzScheduler1) [37bc032f] ++ imageGUID=bbbcf706-d019-4cc5-8096-ddf6d9436553 2017-10-01 21:20:38,060-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (DefaultQuartzScheduler1) [37bc032f] ++ volUUID=6c73cd57-01c6-4541-8157-495e5182f916 2017-10-01 21:20:38,060-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (DefaultQuartzScheduler1) [37bc032f] ++ size=481280 2017-10-01 21:20:39,185-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (DefaultQuartzScheduler1) [37bc032f] FINISH, UploadStreamVDSCommand, log id: 419e99b9 2017-10-01 21:20:39,207-04 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler1) [37bc032f] CommandAsyncTask::Adding CommandMultiAsyncTasks object for command 'ac00a167-2589-4fc1-a824-438d9719c9cf' 2017-10-01 21:20:39,207-04 INFO [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (DefaultQuartzScheduler1) [37bc032f] CommandMultiAsyncTasks::attachTask: Attaching task '153b5c5b-e91f-4f5a-9825-03c287d2dd90' to command 'ac00a167-2589-4fc1-a824-438d9719c9cf'. 2017-10-01 21:20:39,221-04 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler1) [37bc032f] Adding task '153b5c5b-e91f-4f5a-9825-03c287d2dd90' (Parent Command 'UploadStream', Parameters Type 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters'), polling hasn't started yet.. 2017-10-01 21:20:39,227-04 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler1) [37bc032f] BaseAsyncTask::startPollingTask: Starting to poll task '153b5c5b-e91f-4f5a-9825-03c287d2dd90'. 2017-10-01 21:20:39,233-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.UploadStreamCommand] (DefaultQuartzScheduler1) [37bc032f] Lock freed to object 'EngineLock:{exclusiveLocks='', sharedLocks='[45f8b331-842e-48e7-9df8-56adddb93836=VDS_EXECUTION]'}' 2017-10-01 21:20:39,236-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler1) [37bc032f] START, SetVolumeDescriptionVDSCommand( SetVolumeDescriptionVDSCommandParameters:{runAsync='true', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', ignoreFailoverLimit='false', storageDomainId='c8653dba-63ec-4447-872c-0ffcc6af9c0f', imageGroupId='bbbcf706-d019-4cc5-8096-ddf6d9436553', imageId='6c73cd57-01c6-4541-8157-495e5182f916'}), log id: 349f2d25 2017-10-01 21:20:39,236-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler1) [37bc032f] -- executeIrsBrokerCommand: calling 'setVolumeDescription', parameters: 2017-10-01 21:20:39,236-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler1) [37bc032f] ++ spUUID=aca61eac-ed8a-484b-bbee-3645148f6405 2017-10-01 21:20:39,236-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler1) [37bc032f] ++ sdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f 2017-10-01 21:20:39,236-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler1) [37bc032f] ++ imageGroupGUID=bbbcf706-d019-4cc5-8096-ddf6d9436553 2017-10-01 21:20:39,236-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler1) [37bc032f] ++ volUUID=6c73cd57-01c6-4541-8157-495e5182f916 2017-10-01 21:20:39,236-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler1) [37bc032f] ++ description={"Updated":true,"Size":481280,"Last Updated":"Sun Oct 01 21:20:36 EDT 2017","Storage Domains":[{"uuid":"c8653dba-63ec-4447-872c-0ffcc6af9c0f"}],"Disk Description":"OVF_STORE"} 2017-10-01 21:20:39,690-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler1) [37bc032f] FINISH, SetVolumeDescriptionVDSCommand, log id: 349f2d25 2017-10-01 21:20:39,693-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler1) [37bc032f] START, SetVolumeDescriptionVDSCommand( SetVolumeDescriptionVDSCommandParameters:{runAsync='true', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', ignoreFailoverLimit='false', storageDomainId='c8653dba-63ec-4447-872c-0ffcc6af9c0f', imageGroupId='def9e7db-b58b-4df1-ad2b-c47dc60c4303', imageId='357577c1-8d1b-4fad-b470-8e55341e26f6'}), log id: 716e9d8f 2017-10-01 21:20:39,693-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler1) [37bc032f] -- executeIrsBrokerCommand: calling 'setVolumeDescription', parameters: 2017-10-01 21:20:39,693-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler1) [37bc032f] ++ spUUID=aca61eac-ed8a-484b-bbee-3645148f6405 2017-10-01 21:20:39,693-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler1) [37bc032f] ++ sdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f 2017-10-01 21:20:39,693-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler1) [37bc032f] ++ imageGroupGUID=def9e7db-b58b-4df1-ad2b-c47dc60c4303 2017-10-01 21:20:39,693-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler1) [37bc032f] ++ volUUID=357577c1-8d1b-4fad-b470-8e55341e26f6 2017-10-01 21:20:39,693-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler1) [37bc032f] ++ description={"Updated":false,"Last Updated":"Sun Oct 01 17:20:31 EDT 2017","Storage Domains":[{"uuid":"c8653dba-63ec-4447-872c-0ffcc6af9c0f"}],"Disk Description":"OVF_STORE"} 2017-10-01 21:20:39,720-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler1) [37bc032f] FINISH, SetVolumeDescriptionVDSCommand, log id: 716e9d8f 2017-10-01 21:20:39,730-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.UploadStreamCommand] (DefaultQuartzScheduler1) [37bc032f] Lock Acquired to object 'EngineLock:{exclusiveLocks='', sharedLocks='[45f8b331-842e-48e7-9df8-56adddb93836=VDS_EXECUTION]'}' 2017-10-01 21:20:39,745-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.UploadStreamCommand] (DefaultQuartzScheduler1) [37bc032f] Running command: UploadStreamCommand internal: true. Entities affected : ID: c8653dba-63ec-4447-872c-0ffcc6af9c0f Type: Storage 2017-10-01 21:20:39,745-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (DefaultQuartzScheduler1) [37bc032f] START, UploadStreamVDSCommand(HostName = virt1, UploadStreamVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836'}), log id: 336ce152 2017-10-01 21:20:39,745-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (DefaultQuartzScheduler1) [37bc032f] -- executeVdsBrokerCommand, parameters: 2017-10-01 21:20:39,745-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (DefaultQuartzScheduler1) [37bc032f] ++ spUUID=aca61eac-ed8a-484b-bbee-3645148f6405 2017-10-01 21:20:39,745-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (DefaultQuartzScheduler1) [37bc032f] ++ sdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f 2017-10-01 21:20:39,745-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (DefaultQuartzScheduler1) [37bc032f] ++ imageGUID=def9e7db-b58b-4df1-ad2b-c47dc60c4303 2017-10-01 21:20:39,745-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (DefaultQuartzScheduler1) [37bc032f] ++ volUUID=357577c1-8d1b-4fad-b470-8e55341e26f6 2017-10-01 21:20:39,745-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (DefaultQuartzScheduler1) [37bc032f] ++ size=481280 2017-10-01 21:20:40,921-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (DefaultQuartzScheduler1) [37bc032f] FINISH, UploadStreamVDSCommand, log id: 336ce152 2017-10-01 21:20:40,934-04 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler1) [37bc032f] CommandAsyncTask::Adding CommandMultiAsyncTasks object for command '354fba24-94c8-4662-8c77-ee929240eaf4' 2017-10-01 21:20:40,934-04 INFO [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (DefaultQuartzScheduler1) [37bc032f] CommandMultiAsyncTasks::attachTask: Attaching task 'aa1441c3-3bfb-484f-ae45-a1858d7616d2' to command '354fba24-94c8-4662-8c77-ee929240eaf4'. 2017-10-01 21:20:40,946-04 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler1) [37bc032f] Adding task 'aa1441c3-3bfb-484f-ae45-a1858d7616d2' (Parent Command 'UploadStream', Parameters Type 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters'), polling hasn't started yet.. 2017-10-01 21:20:40,951-04 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler1) [37bc032f] BaseAsyncTask::startPollingTask: Starting to poll task 'aa1441c3-3bfb-484f-ae45-a1858d7616d2'. 2017-10-01 21:20:40,956-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.UploadStreamCommand] (DefaultQuartzScheduler1) [37bc032f] Lock freed to object 'EngineLock:{exclusiveLocks='', sharedLocks='[45f8b331-842e-48e7-9df8-56adddb93836=VDS_EXECUTION]'}' 2017-10-01 21:20:40,957-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler1) [37bc032f] START, SetVolumeDescriptionVDSCommand( SetVolumeDescriptionVDSCommandParameters:{runAsync='true', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', ignoreFailoverLimit='false', storageDomainId='c8653dba-63ec-4447-872c-0ffcc6af9c0f', imageGroupId='def9e7db-b58b-4df1-ad2b-c47dc60c4303', imageId='357577c1-8d1b-4fad-b470-8e55341e26f6'}), log id: f65f484 2017-10-01 21:20:41,063-04 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler10) [59f53026] Polling and updating Async Tasks: 2 tasks, 2 tasks to poll now 2017-10-01 21:20:43,098-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler1) [37bc032f] -- executeIrsBrokerCommand: calling 'setVolumeDescription', parameters: 2017-10-01 21:20:43,098-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler1) [37bc032f] ++ spUUID=aca61eac-ed8a-484b-bbee-3645148f6405 2017-10-01 21:20:43,098-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler1) [37bc032f] ++ sdUUID=c8653dba-63ec-4447-872c-0ffcc6af9c0f 2017-10-01 21:20:43,098-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler1) [37bc032f] ++ imageGroupGUID=def9e7db-b58b-4df1-ad2b-c47dc60c4303 2017-10-01 21:20:43,098-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler1) [37bc032f] ++ volUUID=357577c1-8d1b-4fad-b470-8e55341e26f6 2017-10-01 21:20:43,098-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler1) [37bc032f] ++ description={"Updated":true,"Size":481280,"Last Updated":"Sun Oct 01 21:20:36 EDT 2017","Storage Domains":[{"uuid":"c8653dba-63ec-4447-872c-0ffcc6af9c0f"}],"Disk Description":"OVF_STORE"} 2017-10-01 21:20:44,132-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (DefaultQuartzScheduler1) [37bc032f] FINISH, SetVolumeDescriptionVDSCommand, log id: f65f484 2017-10-01 21:20:44,139-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStorageDomainCommand] (DefaultQuartzScheduler1) [37bc032f] Lock freed to object 'EngineLock:{exclusiveLocks='[c8653dba-63ec-4447-872c-0ffcc6af9c0f=STORAGE]', sharedLocks='[aca61eac-ed8a-484b-bbee-3645148f6405=OVF_UPDATE]'}' 2017-10-01 21:20:45,136-04 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler10) [59f53026] SPMAsyncTask::PollTask: Polling task '153b5c5b-e91f-4f5a-9825-03c287d2dd90' (Parent Command 'UploadStream', Parameters Type 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters') returned status 'finished', result 'success'. 2017-10-01 21:20:45,136-04 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler10) [59f53026] BaseAsyncTask::onTaskEndSuccess: Task '153b5c5b-e91f-4f5a-9825-03c287d2dd90' (Parent Command 'UploadStream', Parameters Type 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters') ended successfully. 2017-10-01 21:20:45,138-04 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler10) [59f53026] CommandAsyncTask::endActionIfNecessary: All tasks of command 'ac00a167-2589-4fc1-a824-438d9719c9cf' has ended -> executing 'endAction' 2017-10-01 21:20:45,138-04 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler10) [59f53026] CommandAsyncTask::endAction: Ending action for '1' tasks (command ID: 'ac00a167-2589-4fc1-a824-438d9719c9cf'): calling endAction '. 2017-10-01 21:20:45,138-04 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler10) [59f53026] SPMAsyncTask::PollTask: Polling task 'aa1441c3-3bfb-484f-ae45-a1858d7616d2' (Parent Command 'UploadStream', Parameters Type 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters') returned status 'finished', result 'success'. 2017-10-01 21:20:45,138-04 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (DefaultQuartzScheduler10) [59f53026] BaseAsyncTask::onTaskEndSuccess: Task 'aa1441c3-3bfb-484f-ae45-a1858d7616d2' (Parent Command 'UploadStream', Parameters Type 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters') ended successfully. 2017-10-01 21:20:45,138-04 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-6-thread-45) [59f53026] CommandAsyncTask::endCommandAction [within thread] context: Attempting to endAction 'UploadStream', 2017-10-01 21:20:45,139-04 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler10) [59f53026] CommandAsyncTask::endActionIfNecessary: All tasks of command '354fba24-94c8-4662-8c77-ee929240eaf4' has ended -> executing 'endAction' 2017-10-01 21:20:45,139-04 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (DefaultQuartzScheduler10) [59f53026] CommandAsyncTask::endAction: Ending action for '1' tasks (command ID: '354fba24-94c8-4662-8c77-ee929240eaf4'): calling endAction '. 2017-10-01 21:20:45,139-04 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-6-thread-16) [59f53026] CommandAsyncTask::endCommandAction [within thread] context: Attempting to endAction 'UploadStream', 2017-10-01 21:20:45,142-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.UploadStreamCommand] (org.ovirt.thread.pool-6-thread-45) [37bc032f] Ending command 'org.ovirt.engine.core.bll.storage.ovfstore.UploadStreamCommand' successfully. 2017-10-01 21:20:45,143-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.UploadStreamCommand] (org.ovirt.thread.pool-6-thread-16) [37bc032f] Ending command 'org.ovirt.engine.core.bll.storage.ovfstore.UploadStreamCommand' successfully. 2017-10-01 21:20:45,147-04 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-6-thread-45) [37bc032f] CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type 'UploadStream' completed, handling the result. 2017-10-01 21:20:45,147-04 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-6-thread-45) [37bc032f] CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type 'UploadStream' succeeded, clearing tasks. 2017-10-01 21:20:45,147-04 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-6-thread-45) [37bc032f] SPMAsyncTask::ClearAsyncTask: Attempting to clear task '153b5c5b-e91f-4f5a-9825-03c287d2dd90' 2017-10-01 21:20:45,147-04 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-6-thread-16) [37bc032f] CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type 'UploadStream' completed, handling the result. 2017-10-01 21:20:45,147-04 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-6-thread-16) [37bc032f] CommandAsyncTask::HandleEndActionResult [within thread]: endAction for action type 'UploadStream' succeeded, clearing tasks. 2017-10-01 21:20:45,147-04 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-6-thread-16) [37bc032f] SPMAsyncTask::ClearAsyncTask: Attempting to clear task 'aa1441c3-3bfb-484f-ae45-a1858d7616d2' 2017-10-01 21:20:45,147-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-6-thread-45) [37bc032f] START, SPMClearTaskVDSCommand( SPMTaskGuidBaseVDSCommandParameters:{runAsync='true', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', ignoreFailoverLimit='false', taskId='153b5c5b-e91f-4f5a-9825-03c287d2dd90'}), log id: 50b3d3c6 2017-10-01 21:20:45,147-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-6-thread-45) [37bc032f] START, HSMClearTaskVDSCommand(HostName = virt1, HSMTaskGuidBaseVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', taskId='153b5c5b-e91f-4f5a-9825-03c287d2dd90'}), log id: 2cbd09aa 2017-10-01 21:20:45,147-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-6-thread-16) [37bc032f] START, SPMClearTaskVDSCommand( SPMTaskGuidBaseVDSCommandParameters:{runAsync='true', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', ignoreFailoverLimit='false', taskId='aa1441c3-3bfb-484f-ae45-a1858d7616d2'}), log id: 7efe3013 2017-10-01 21:20:45,983-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-6-thread-45) [37bc032f] FINISH, HSMClearTaskVDSCommand, log id: 2cbd09aa 2017-10-01 21:20:45,983-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-6-thread-45) [37bc032f] FINISH, SPMClearTaskVDSCommand, log id: 50b3d3c6 2017-10-01 21:20:45,983-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-6-thread-16) [37bc032f] START, HSMClearTaskVDSCommand(HostName = virt1, HSMTaskGuidBaseVDSCommandParameters:{runAsync='true', hostId='45f8b331-842e-48e7-9df8-56adddb93836', taskId='aa1441c3-3bfb-484f-ae45-a1858d7616d2'}), log id: 349a6769 2017-10-01 21:20:45,987-04 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-6-thread-45) [37bc032f] BaseAsyncTask::removeTaskFromDB: Removed task '153b5c5b-e91f-4f5a-9825-03c287d2dd90' from DataBase 2017-10-01 21:20:45,987-04 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-6-thread-45) [37bc032f] CommandAsyncTask::HandleEndActionResult [within thread]: Removing CommandMultiAsyncTasks object for entity 'ac00a167-2589-4fc1-a824-438d9719c9cf' 2017-10-01 21:20:46,994-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (org.ovirt.thread.pool-6-thread-16) [37bc032f] FINISH, HSMClearTaskVDSCommand, log id: 349a6769 2017-10-01 21:20:46,994-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (org.ovirt.thread.pool-6-thread-16) [37bc032f] FINISH, SPMClearTaskVDSCommand, log id: 7efe3013 2017-10-01 21:20:46,996-04 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (org.ovirt.thread.pool-6-thread-16) [37bc032f] BaseAsyncTask::removeTaskFromDB: Removed task 'aa1441c3-3bfb-484f-ae45-a1858d7616d2' from DataBase 2017-10-01 21:20:46,996-04 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (org.ovirt.thread.pool-6-thread-16) [37bc032f] CommandAsyncTask::HandleEndActionResult [within thread]: Removing CommandMultiAsyncTasks object for entity '354fba24-94c8-4662-8c77-ee929240eaf4' 2017-10-01 21:20:51,116-04 INFO [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] (DefaultQuartzScheduler10) [37bc032f] Command 'ProcessOvfUpdateForStorageDomain' id: '734de656-38a7-4022-87c8-2fda5545d2a5' child commands '[ac00a167-2589-4fc1-a824-438d9719c9cf, 354fba24-94c8-4662-8c77-ee929240eaf4]' executions were completed, status 'SUCCEEDED' 2017-10-01 21:20:52,140-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStorageDomainCommand] (DefaultQuartzScheduler1) [37bc032f] Ending command 'org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStorageDomainCommand' successfully. 2017-10-01 21:21:03,070-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [6729a20b] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 21:21:59,361-04 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler3) [359e9e94] Setting new tasks map. The map contains now 0 tasks 2017-10-01 21:21:59,361-04 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (DefaultQuartzScheduler3) [359e9e94] Cleared all tasks of pool 'aca61eac-ed8a-484b-bbee-3645148f6405'. 2017-10-01 21:22:03,090-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [7b53d93a] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 21:23:03,114-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [6f89f609] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 21:24:03,134-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [15ff9782] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 21:25:03,153-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [3a026e41] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 21:26:03,174-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler7) [2008d428] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 21:27:03,195-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [63e5a21f] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 21:28:03,219-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [435c687c] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 21:28:32,752-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-6-thread-42) [36b44955] START, GetFileStatsVDSCommand( GetFileStatsParameters:{runAsync='true', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', ignoreFailoverLimit='false'}), log id: 33ffea8e 2017-10-01 21:28:35,761-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-6-thread-42) [36b44955] FINISH, GetFileStatsVDSCommand, return: {virtio-win-0.1.126.iso={status=0, ctime=1497024605.0, size=155856896}, server2012r2_virtio.iso={status=0, ctime=1497297085.0, size=4977590272}, en_windows_7_professional_with_sp1_x64_dvd_621750.iso={status=0, ctime=1435288543.0, size=3319478272}, virtio-win-0.1.105.iso={status=0, ctime=1435288648.0, size=56584192}}, log id: 33ffea8e 2017-10-01 21:28:35,764-04 INFO [org.ovirt.engine.core.bll.storage.domain.IsoDomainListSynchronizer] (org.ovirt.thread.pool-6-thread-42) [36b44955] Finished automatic refresh process for 'ISO' file type with success, for storage domain id '63d680c7-4c94-4b29-a6cb-f288a128b9bc'. 2017-10-01 21:29:03,239-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [7403585e] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 21:30:03,258-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [724f7683] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 21:31:03,279-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [bc3c86d] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 21:32:03,299-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [1546800b] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 21:33:03,320-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [1ab74b4e] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 21:34:03,340-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [1a1aa83] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 21:35:03,359-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [a43b503] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 21:36:03,382-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [24bea598] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 21:37:03,402-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [2b844bd0] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 21:38:03,426-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [49905c8f] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 21:39:03,445-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [5fda64f6] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 21:40:03,465-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [27291e66] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 21:41:03,486-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [52c4789a] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 21:42:03,511-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [438a2a05] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 21:43:03,536-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [236c8b93] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 21:44:03,555-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [560bb690] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 21:45:03,574-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [27e91b7f] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 21:46:03,597-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [7c6fcb08] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 21:47:03,616-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [6c33a5b7] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 21:48:03,641-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [475733] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 21:49:03,666-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [1740e047] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 21:50:03,687-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [48b2ab55] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 21:51:03,708-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [4346a54b] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 21:52:03,727-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [b6edf2b] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 21:53:03,751-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [54cda329] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 21:54:03,770-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [206a63bc] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 21:55:03,789-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [5891fc39] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 21:56:03,807-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [501fbe8f] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 21:57:03,827-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [acc1ee2] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 21:58:03,852-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [360fd05b] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 21:59:03,872-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [60ce4c1b] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 22:00:03,890-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [a30a40a] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 22:01:03,912-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [581eba18] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 22:02:03,933-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [292e975c] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 22:03:03,960-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [4006b939] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 22:04:03,980-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [35e5a4b0] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 22:05:04,001-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [70e80fc5] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 22:06:04,021-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [6edd67eb] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 22:07:04,040-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [3807500f] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 22:08:04,066-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [6f85ba69] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 22:09:04,090-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [75460dec] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 22:10:04,109-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [170ee109] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 22:11:04,129-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler7) [33577f83] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 22:12:04,148-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [117f310c] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 22:13:04,170-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [747c7eff] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 22:14:04,189-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [549a23a4] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 22:15:04,208-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [46c452d3] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 22:16:04,227-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [1d24f24e] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 22:17:04,247-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [7049c418] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 22:18:04,271-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [2cb125a3] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 22:19:04,290-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler7) [1d86bb1b] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 22:20:04,312-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [65536d2a] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 22:20:44,139-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.OvfDataUpdater] (DefaultQuartzScheduler2) [ac793b0] Attempting to update VMs/Templates Ovf. 2017-10-01 22:20:44,142-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [46801f39] Before acquiring and wait lock 'EngineLock:{exclusiveLocks='[aca61eac-ed8a-484b-bbee-3645148f6405=OVF_UPDATE]', sharedLocks=''}' 2017-10-01 22:20:44,142-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [46801f39] Lock-wait acquired to object 'EngineLock:{exclusiveLocks='[aca61eac-ed8a-484b-bbee-3645148f6405=OVF_UPDATE]', sharedLocks=''}' 2017-10-01 22:20:44,143-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [46801f39] Running command: ProcessOvfUpdateForStoragePoolCommand internal: true. Entities affected : ID: aca61eac-ed8a-484b-bbee-3645148f6405 Type: StoragePool 2017-10-01 22:20:44,205-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [46801f39] Attempting to update VM OVFs in Data Center 'EECS-Primary' 2017-10-01 22:20:44,210-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [46801f39] Successfully updated VM OVFs in Data Center 'EECS-Primary' 2017-10-01 22:20:44,210-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [46801f39] Attempting to update template OVFs in Data Center 'EECS-Primary' 2017-10-01 22:20:44,211-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [46801f39] Successfully updated templates OVFs in Data Center 'EECS-Primary' 2017-10-01 22:20:44,211-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [46801f39] Attempting to remove unneeded template/vm OVFs in Data Center 'EECS-Primary' 2017-10-01 22:20:44,213-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [46801f39] Successfully removed unneeded template/vm OVFs in Data Center 'EECS-Primary' 2017-10-01 22:20:44,214-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler2) [46801f39] Lock freed to object 'EngineLock:{exclusiveLocks='[aca61eac-ed8a-484b-bbee-3645148f6405=OVF_UPDATE]', sharedLocks=''}' 2017-10-01 22:21:04,332-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [13b9030d] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 22:22:04,351-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [3b2eb32d] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 22:23:04,375-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [32899225] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 22:24:04,395-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [1c6416bf] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 22:25:04,416-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [203aa7a6] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 22:26:04,436-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [3e6d2f82] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 22:27:04,456-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [b28f709] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 22:28:04,480-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [166b53cb] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 22:28:35,767-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-6-thread-43) [59363579] START, GetFileStatsVDSCommand( GetFileStatsParameters:{runAsync='true', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', ignoreFailoverLimit='false'}), log id: 1171f21 2017-10-01 22:28:37,721-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-6-thread-43) [59363579] FINISH, GetFileStatsVDSCommand, return: {virtio-win-0.1.126.iso={status=0, ctime=1497024605.0, size=155856896}, server2012r2_virtio.iso={status=0, ctime=1497297085.0, size=4977590272}, en_windows_7_professional_with_sp1_x64_dvd_621750.iso={status=0, ctime=1435288543.0, size=3319478272}, virtio-win-0.1.105.iso={status=0, ctime=1435288648.0, size=56584192}}, log id: 1171f21 2017-10-01 22:28:37,725-04 INFO [org.ovirt.engine.core.bll.storage.domain.IsoDomainListSynchronizer] (org.ovirt.thread.pool-6-thread-43) [59363579] Finished automatic refresh process for 'ISO' file type with success, for storage domain id '63d680c7-4c94-4b29-a6cb-f288a128b9bc'. 2017-10-01 22:29:04,499-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [15ff930a] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 22:30:04,519-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [6ba84c7b] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 22:31:04,539-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [64234e0b] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 22:32:04,558-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [3d87c0b1] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 22:33:04,582-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [418c37c0] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 22:34:04,604-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [3e087040] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 22:35:04,626-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [6e8da22d] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 22:36:04,644-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [17598d24] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 22:37:04,663-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [1d33e22] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 22:38:04,687-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [6d7ac2d1] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 22:39:04,705-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [3ae5795d] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 22:40:04,725-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [187e9e13] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 22:41:04,746-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [a2718d9] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 22:42:04,777-04 INFO [org.ovirt.engine.core.sso.servlets.OAuthRevokeServlet] (default task-251) [] User jas@eecs.yorku.ca successfully logged out 2017-10-01 22:42:04,791-04 INFO [org.ovirt.engine.core.bll.aaa.TerminateSessionsForTokenCommand] (default task-241) [2e0caebc] Running command: TerminateSessionsForTokenCommand internal: true. 2017-10-01 22:42:04,796-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [160f76b0] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 22:43:04,820-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [40eb2ce3] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 22:44:04,840-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [21bc4a96] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 22:45:04,861-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [71186c21] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 22:46:04,880-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [4bc423ba] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 22:47:04,900-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [193847a9] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 22:48:04,925-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [e8d811b] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 22:49:04,947-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [63a92bb3] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 22:50:04,969-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [79dbfbd3] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 22:51:04,989-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [4b1d2862] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 22:52:05,008-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [3d997b98] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 22:53:05,033-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [7dcef949] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 22:54:05,054-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [a53840b] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 22:55:05,075-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [2ec8f5a5] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 22:56:05,102-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [6b443e27] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 22:57:05,119-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [6f0a320] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 22:58:05,144-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [3881586e] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 22:59:05,165-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [6f982b3c] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 23:00:05,184-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [63b9f250] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 23:01:05,202-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [5bb708ee] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 23:02:05,224-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [6f56bbbc] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 23:03:05,248-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [217bebe3] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 23:04:05,270-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [7a1f29c6] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 23:05:05,291-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [21c69590] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 23:06:05,312-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler7) [521db62a] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 23:07:05,332-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [38a1f666] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 23:08:05,356-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [708f51f] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 23:09:05,377-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [177ab0d3] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 23:10:05,397-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [dd2d8ff] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 23:11:05,417-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [1a54d942] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 23:12:05,437-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [2642849e] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 23:13:05,462-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [27640baf] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 23:14:05,481-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [89045] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 23:15:05,500-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [7339244e] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 23:16:05,519-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler7) [7daf9a0d] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 23:17:05,539-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [c7ebc86] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 23:18:05,566-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [3c0c5d9f] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 23:19:05,586-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [2bca085b] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 23:20:05,605-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [139998f4] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 23:20:44,214-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.OvfDataUpdater] (DefaultQuartzScheduler3) [578fd91c] Attempting to update VMs/Templates Ovf. 2017-10-01 23:20:44,217-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler3) [1b911af6] Before acquiring and wait lock 'EngineLock:{exclusiveLocks='[aca61eac-ed8a-484b-bbee-3645148f6405=OVF_UPDATE]', sharedLocks=''}' 2017-10-01 23:20:44,217-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler3) [1b911af6] Lock-wait acquired to object 'EngineLock:{exclusiveLocks='[aca61eac-ed8a-484b-bbee-3645148f6405=OVF_UPDATE]', sharedLocks=''}' 2017-10-01 23:20:44,218-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler3) [1b911af6] Running command: ProcessOvfUpdateForStoragePoolCommand internal: true. Entities affected : ID: aca61eac-ed8a-484b-bbee-3645148f6405 Type: StoragePool 2017-10-01 23:20:44,223-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler3) [1b911af6] Attempting to update VM OVFs in Data Center 'EECS-Primary' 2017-10-01 23:20:44,228-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler3) [1b911af6] Successfully updated VM OVFs in Data Center 'EECS-Primary' 2017-10-01 23:20:44,228-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler3) [1b911af6] Attempting to update template OVFs in Data Center 'EECS-Primary' 2017-10-01 23:20:44,228-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler3) [1b911af6] Successfully updated templates OVFs in Data Center 'EECS-Primary' 2017-10-01 23:20:44,228-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler3) [1b911af6] Attempting to remove unneeded template/vm OVFs in Data Center 'EECS-Primary' 2017-10-01 23:20:44,229-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler3) [1b911af6] Successfully removed unneeded template/vm OVFs in Data Center 'EECS-Primary' 2017-10-01 23:20:44,230-04 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler3) [1b911af6] Lock freed to object 'EngineLock:{exclusiveLocks='[aca61eac-ed8a-484b-bbee-3645148f6405=OVF_UPDATE]', sharedLocks=''}' 2017-10-01 23:21:05,628-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [3ae366f5] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 23:22:05,648-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [2aac3d93] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 23:23:05,674-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [61d251d6] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 23:24:05,693-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler7) [ca319ee] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 23:25:05,713-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [14312c2a] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 23:26:05,732-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [f6e42ef] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 23:27:05,753-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [7ba5c031] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 23:28:05,777-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [13de6c0f] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 23:28:37,725-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-6-thread-4) [125be5ed] START, GetFileStatsVDSCommand( GetFileStatsParameters:{runAsync='true', storagePoolId='aca61eac-ed8a-484b-bbee-3645148f6405', ignoreFailoverLimit='false'}), log id: 292ae6cc 2017-10-01 23:28:38,643-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-6-thread-4) [125be5ed] FINISH, GetFileStatsVDSCommand, return: {virtio-win-0.1.126.iso={status=0, ctime=1497024605.0, size=155856896}, server2012r2_virtio.iso={status=0, ctime=1497297085.0, size=4977590272}, en_windows_7_professional_with_sp1_x64_dvd_621750.iso={status=0, ctime=1435288543.0, size=3319478272}, virtio-win-0.1.105.iso={status=0, ctime=1435288648.0, size=56584192}}, log id: 292ae6cc 2017-10-01 23:28:38,647-04 INFO [org.ovirt.engine.core.bll.storage.domain.IsoDomainListSynchronizer] (org.ovirt.thread.pool-6-thread-4) [125be5ed] Finished automatic refresh process for 'ISO' file type with success, for storage domain id '63d680c7-4c94-4b29-a6cb-f288a128b9bc'. 2017-10-01 23:29:05,797-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [106f144b] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 23:30:05,820-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [1b329ec1] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 23:31:05,840-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [3d5d1320] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 23:32:05,859-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [18cd16e4] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 23:33:05,883-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [5eb11b8c] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 23:34:05,902-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [44c89014] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 23:35:05,928-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [4ccba765] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 23:36:05,948-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [6b22fee1] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 23:37:05,969-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [570c63db] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 23:38:05,994-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler3) [774a26a9] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 23:39:06,013-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [732f85b9] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 23:40:06,034-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [54b52cba] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 23:41:06,053-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [32c25617] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 23:42:06,073-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler2) [192ad55b] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 23:43:06,097-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [7bdd9e12] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 23:44:06,122-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler5) [265a306b] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 23:45:06,142-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [5af5d747] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 23:46:06,162-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [16ef72ea] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 23:47:06,184-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [685c2648] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 23:48:06,208-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [1aa3e4b1] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 23:49:06,227-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [62c4d9e7] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 23:50:06,244-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [5b18cc87] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 23:51:06,264-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler9) [744fcc67] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 23:52:06,287-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler8) [f4c8207] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 23:53:06,311-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [1b4b8147] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 23:54:06,331-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler10) [55f3caf3] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 23:55:06,350-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler4) [391069f0] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 23:56:06,369-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler1) [4bc92eec] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 23:57:06,389-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [5aca02e] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 23:58:06,415-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler7) [94b7a70] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'. 2017-10-01 23:59:06,434-04 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (DefaultQuartzScheduler6) [652e043d] Not removing session 'GfGyhEQ8bDHclSIjImnT647YKJBIbgE5r+vgf6SN+mP/YbU0P5ba3y83zxFOnS8YCkPwgX2MvdwLazoGwnJFoA==', session has running commands for user 'jas@eecs.yorku.ca-authz'.