[2020-09-21 07:50:02]
         fs1  fs3  fs5  fs6
1  Gig   UP   UP   UP   UP 
10 Gig   n/a  UP   UP   UP 
IB       n/a  UP   UP   UP 
[2020-09-21 07:50:02]

 | 3-11|   OOOOOOOO | 3-21|   ........ | 3-31|   ........ | 3-41|   ........ | 3-51|   OOOOOOOO | 3-61|   OOOOOOOO | 3-71|   OOOOO...
 | 3-12|   O....... | 3-22|   ........ | 3-32|   ........ | 3-42|   ........ | 3-52|   OOOOOOOO | 3-62|   OOOOOOOO | 3-72|     DOWN  
 | 3-13|     DOWN   | 3-23|   ........ | 3-33|   ........ | 3-43|   ........ | 3-53|   OOOOOOOO | 3-63|   OOOOOOOO | 3-73|   OOOOOOO.
 | 3-14|   ........ | 3-24|   ........ | 3-34|   ........ | 3-44|   ........ | 3-54|   OOOOOOOO | 3-64|   OOOOOOOO | 3-74|   OOOOOOOO
 | 3-15|   ........ | 3-25|     DOWN   | 3-35|   ........ | 3-45|   OOOOOOOO | 3-55|   OOOOOOOO | 3-65|   OOOOOOOO | 3-75|   OOOOOOO.
 | 3-16|   ........ | 3-26|   ........ | 3-36|   ........ | 3-46|   OO...... | 3-56|   OOOOOOOO | 3-66|     DOWN   | 3-76|     DOWN  
 | 3-17|   ........ | 3-27|   ........ | 3-37|   ........ | 3-47|   O....... | 3-57|   OOOOOOOO | 3-67|   OOOOOOOO | 3-77|   OOOOOOOO
 | 3-18|   ........ | 3-28|   ........ | 3-38|   ........ | 3-48|   OOOOOOOO | 3-58|   OOOOOOOO | 3-68|   OOOOOOOO | 3-78|   ........


 | 4-11|   OOOOO... | 4-21|   OOOOO... | 4-31|   OOOOO... | 4-41|   OOOOO... | 4-51|   OOOOO... | 4-61|   OOOOOO.. | 4-71|   OOOOO... | 4-81|   ........
 | 4-12|   OOOOO... | 4-22|   OOOOO... | 4-32|   OOOOO... | 4-42|   OOOOO... | 4-52|   OOOOOOO. | 4-62|   OOOOO... | 4-72|   OOOOO... | 4-82|   ........
 | 4-13|   OOOOO... | 4-23|   OOOOO... | 4-33|   OOOOO... | 4-43|   OOOOO... | 4-53|   OOOOO... | 4-63|     DOWN   | 4-73|   OOOOO... | 4-83|   ........
 | 4-14|   OOOOO... | 4-24|   OOOOO... | 4-34|   OOOOO... | 4-44|   OOOOO... | 4-54|   OOOOO... | 4-64|   OOOOO... | 4-74|   OOOOO...                   


 | 5- 1|   OOOOOOOO | 5-11|   OOOOO... | 5-21|   ........
 | 5- 2|   OOOOO... | 5-12|   OOOOO... | 5-22|   ........
 | 5- 3|   OOOOO... | 5-13|   OOOOO... | 5-23|   ........
 | 5- 4|   OOOOO... | 5-14|   OOOOO... | 5-24|     DOWN  
 | 5- 5|   OOOOO... | 5-15|   OOOOO... | 5-25|     DOWN  
 | 5- 6|   OOOO.... | 5-16|   ........ | 5-26|   ........
 | 5- 7|   OOOOO... | 5-17|   ........ | 5-27|   ........
 | 5- 8|   OOOO.... | 5-18|   ........ | 5-28|   ........
 | 5- 9|   OOOO.... | 5-19|     DOWN   | 5-29|   ........
 | 5-10|   OOOOO... | 5-20|     DOWN   | 5-30|   ........


 | 9- 1|   ........
 | 9- 2|   ........
 | 9- 3|   ........


 |10- 1|   ........
 |10- 2|     DOWN  


 |21-11|   ........ |21-21|   ........ |21-31|   ........ |21-41|   ........ |21-51|   ........ |21-61|   ........ |21-71|   ........ |21-81|   ........ |21-91|     DOWN  
 |21-12|   ........ |21-22|   ........ |21-32|   ........ |21-42|   ........ |21-52|   ........ |21-62|   OOOOOOOO |21-72|   ........ |21-82|   ........ |21-92|     DOWN  
 |21-13|   ........ |21-23|   ........ |21-33|   ........ |21-43|   ........ |21-53|   ........ |21-63|   ........ |21-73|   ........ |21-83|   ........ |21-93|     DOWN  
 |21-14|   ........ |21-24|   ........ |21-34|   ........ |21-44|   ........ |21-54|   ........ |21-64|   ........ |21-74|   ........ |21-84|   ........ |21-94|     DOWN  


 |22-11|   OOOOOOO. |22-21|   O....... |22-31|   ........ |22-41|   ........ |22-51|     DOWN  
 |22-12|   O....... |22-22|   O....... |22-32|     DOWN   |22-42|   ........ |22-52|     DOWN  
 |22-13|   O....... |22-23|   O....... |22-33|   OOOOOOOO |22-43|   ........ |22-53|   OOOOOO..
 |22-14|   O....... |22-24|   ........ |22-34|   OOOOOO.. |22-44|   OOOOOO.. |22-54|   OOOOOO..


 |91- 1|   ........
 |91- 2|   ........
 |91- 3|   ........
 |91- 4|   ........
 |91- 5|   ........
 |91- 6|   ........


Totals:
Total cores: 6568 
Total memory: 58.03 TiB 

Down:
Down cores: 732 ( 11.14%) 
Down memory: 6.19 TiB ( 10.67%) 

Used:
Used cores: 1578 ( 24.03%) 
Used memory: 6.50 TiB ( 11.21%) 

Utilization excluding down
CPU utilization:  27.04% (global)  34.80% (average per node)
Memory utilization:  12.55% (global)  15.95% (average per node)

Average per node utilization (excluding down):  35.68% 
[2020-09-21 07:50:02]
s03n13 health-check: s03n13 25-Aug-2020 10:01 Lost part of memory 
s03n25 tries to pxeboot in a loop - dead systemdisk? 
s03n66 tries to pxeboot in a loop - dead systemdisk? 
s03n72 fails in pxeboot - /dev/root does not exist 
s03n76 tries to pxeboot in a loop - dead systemdisk? 
s04n63 Prolog error 
s05n19 pon and pof doesn't work 
s05n20 pon and pof doesn't work 
s05n24 pon and pof doesn't work 
s05n25 Not responding 
s10n02 health-check: df /faststorage hangs 
s21n91 fan malfunction 
s21n92 fan malfunction 
s21n93 fan malfunction 
s21n94 fan malfunction 
s22n32 Not responding 
s22n51 has issues 
s22n52 potential memory errors 

[2020-09-21 07:50:14]
[store03 s94n01] Controller 0
[store03 s94n01] Virtual Drive  0 - RAID1 - UU - Optimal
[store03 s94n01] 

[store03 s94n02] Controller 0
[store03 s94n02] Virtual Drive  0 - RAID1 - UU - Optimal
[store03 s94n02] 

[store03 s95n01] Controller 0
[store03 s95n01] Virtual Drive  0 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n01] Virtual Drive  1 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n01] Virtual Drive  2 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n01] Virtual Drive  3 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n01] Virtual Drive  4 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n01] Virtual Drive  5 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n01] Virtual Drive  6 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n01] Controller 1
[store03 s95n01] Virtual Drive  0 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n01] Virtual Drive  1 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n01] Virtual Drive  2 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n01] Virtual Drive  3 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n01] Virtual Drive  4 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n01] Virtual Drive  5 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n01] Virtual Drive  6 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n01] 

[store03 s95n02] Controller 0
[store03 s95n02] Virtual Drive  0 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n02] Virtual Drive  1 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n02] Virtual Drive  2 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n02] Virtual Drive  3 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n02] Virtual Drive  4 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n02] Virtual Drive  5 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n02] Virtual Drive  6 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n02] Virtual Drive  7 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n02] Virtual Drive  8 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n02] Virtual Drive  9 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n02] Virtual Drive 10 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n02] Virtual Drive 11 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n02] Virtual Drive 12 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n02] Virtual Drive 13 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n02] 

[store03 s96n01] Controller 0
[store03 s96n01] Virtual Drive  7 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n01] Virtual Drive  8 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n01] Virtual Drive  9 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n01] Virtual Drive 10 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n01] Virtual Drive 11 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n01] Virtual Drive 12 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n01] Virtual Drive 13 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n01] Controller 1
[store03 s96n01] Virtual Drive  0 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n01] Virtual Drive  1 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n01] Virtual Drive  2 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n01] Virtual Drive  3 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n01] Virtual Drive  4 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n01] Virtual Drive  5 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n01] Virtual Drive  6 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n01] 

[store03 s96n02] Controller 0
[store03 s96n02] Virtual Drive  0 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n02] Virtual Drive  1 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n02] Virtual Drive  2 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n02] Virtual Drive  3 - RAID6 - U_UUUUUUUUUU - Partially Degraded - 
[store03 s96n02] Virtual Drive  4 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n02] Virtual Drive  5 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n02] Virtual Drive  6 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n02] Virtual Drive  7 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n02] Virtual Drive  8 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n02] Virtual Drive  9 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n02] Virtual Drive 10 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n02] Virtual Drive 11 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n02] Virtual Drive 12 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n02] Virtual Drive 13 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n02] 

[store03 s97n01] Controller 0
[store03 s97n01] Virtual Drive  0 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n01] Virtual Drive  1 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n01] Virtual Drive  2 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n01] Virtual Drive  3 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n01] Virtual Drive  4 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n01] Virtual Drive  5 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n01] Virtual Drive  6 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n01] Controller 1
[store03 s97n01] Virtual Drive  0 - RAID1 - UU - Optimal
[store03 s97n01] Controller 2
[store03 s97n01] Virtual Drive  0 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n01] Virtual Drive  1 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n01] Virtual Drive  2 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n01] Virtual Drive  3 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n01] Virtual Drive  4 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n01] Virtual Drive  5 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n01] Virtual Drive  6 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n01] 

[store03 s97n02] Controller 0
[store03 s97n02] Virtual Drive  0 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n02] Virtual Drive  1 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n02] Virtual Drive  2 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n02] Virtual Drive  3 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n02] Virtual Drive  4 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n02] Virtual Drive  5 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n02] Virtual Drive  6 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n02] Controller 1
[store03 s97n02] Virtual Drive  0 - RAID1 - UU - Optimal
[store03 s97n02] Controller 2
[store03 s97n02] Virtual Drive  0 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n02] Virtual Drive  1 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n02] Virtual Drive  2 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n02] Virtual Drive  3 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n02] Virtual Drive  4 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n02] Virtual Drive  5 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n02] Virtual Drive  6 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n02] 

[store03 s98n01] Controller 0
[store03 s98n01] Virtual Drive  0 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n01] Virtual Drive  1 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n01] Virtual Drive  2 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n01] Virtual Drive  3 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n01] Virtual Drive  4 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n01] Virtual Drive  5 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n01] Virtual Drive  6 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n01] Controller 1
[store03 s98n01] Virtual Drive  0 - RAID1 - UU - Optimal
[store03 s98n01] Controller 2
[store03 s98n01] Virtual Drive  0 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n01] Virtual Drive  1 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n01] Virtual Drive  2 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n01] Virtual Drive  3 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n01] Virtual Drive  4 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n01] Virtual Drive  5 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n01] Virtual Drive  6 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n01] 

[store03 s98n02] Controller 0
[store03 s98n02] Virtual Drive  0 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n02] Virtual Drive  1 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n02] Virtual Drive  2 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n02] Virtual Drive  3 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n02] Virtual Drive  4 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n02] Virtual Drive  5 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n02] Virtual Drive  6 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n02] Controller 1
[store03 s98n02] Virtual Drive  0 - RAID1 - UU - Optimal
[store03 s98n02] Controller 2
[store03 s98n02] Virtual Drive  0 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n02] Virtual Drive  1 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n02] Virtual Drive  2 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n02] Virtual Drive  3 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n02] Virtual Drive  4 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n02] Virtual Drive  5 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n02] Virtual Drive  6 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n02] 

[2020-09-21 07:45:04]
    NodeID   TargetID         Pool           Total            Used            Free   Use% Usage Bar                    ITotal        IFree     %
    ======   ========         ====           =====            ====            ====   ==== =========                    ======        =====     =
      9501       3001    emergency       80008.1GB       65785.3GB       14222.8GB    82% [################    ]      1490.3M      1487.6M  100%
      9501       3002       normal       80008.1GB       67235.6GB       12772.5GB    84% [#################   ]      1490.3M      1487.2M  100%
      9501       3003       normal       80008.1GB       67603.9GB       12404.2GB    84% [#################   ]      1490.3M      1487.2M  100%
      9501       3004       normal       80008.1GB       68252.5GB       11755.6GB    85% [#################   ]      1490.3M      1487.2M  100%
      9501       3005       normal       80008.1GB       65840.4GB       14167.7GB    82% [################    ]      1490.3M      1487.2M  100%
      9501       3006       normal       80008.1GB       67903.9GB       12104.2GB    85% [#################   ]      1490.3M      1487.2M  100%
      9501       3007       normal       80008.1GB       69374.6GB       10633.4GB    87% [#################   ]      1490.3M      1487.3M  100%
      9501       3008       normal       80008.1GB       68445.6GB       11562.5GB    86% [#################   ]      1490.3M      1487.1M  100%
      9501       3009       normal       80008.1GB       68373.4GB       11634.7GB    85% [#################   ]      1490.3M      1487.1M  100%
      9501       3010       normal       80008.1GB       66420.9GB       13587.1GB    83% [#################   ]      1490.3M      1487.1M  100%
      9501       3011       normal       80008.1GB       64446.8GB       15561.3GB    81% [################    ]      1490.3M      1487.2M  100%
      9501       3012       normal       80008.1GB       66639.0GB       13369.0GB    83% [#################   ]      1490.3M      1487.2M  100%
      9501       3013       normal       80008.1GB       65312.1GB       14695.9GB    82% [################    ]      1490.3M      1487.1M  100%
      9502       3014       normal       80008.1GB       64608.1GB       15400.0GB    81% [################    ]      1490.3M      1487.1M  100%
      9502       3015       normal       80008.1GB       66251.0GB       13757.1GB    83% [#################   ]      1490.3M      1487.1M  100%
      9502       3016       normal       80008.1GB       70624.2GB        9383.9GB    88% [##################  ]      1490.3M      1489.5M  100%
      9502       3017       normal       80008.1GB       66390.6GB       13617.4GB    83% [#################   ]      1490.3M      1487.1M  100%
      9502       3018       normal       80008.1GB       67851.1GB       12156.9GB    85% [#################   ]      1490.3M      1487.1M  100%
      9502       3019       normal       80008.1GB       65833.2GB       14174.9GB    82% [################    ]      1490.3M      1487.1M  100%
      9502       3020       normal       80008.1GB       67148.2GB       12859.8GB    84% [#################   ]      1490.3M      1487.1M  100%
      9601       3021       normal       80008.1GB       66758.0GB       13250.0GB    83% [#################   ]      1490.3M      1487.1M  100%
      9601       3022       normal       80008.1GB       66349.9GB       13658.2GB    83% [#################   ]      1490.3M      1487.1M  100%
      9601       3023       normal       80008.1GB       66587.6GB       13420.5GB    83% [#################   ]      1490.3M      1487.2M  100%
      9601       3024       normal       80008.1GB       68109.2GB       11898.9GB    85% [#################   ]      1490.3M      1487.1M  100%
      9601       3025       normal       80008.1GB       66073.5GB       13934.6GB    83% [#################   ]      1490.3M      1487.1M  100%
      9601       3026       normal       80008.1GB       64096.1GB       15911.9GB    80% [################    ]      1490.3M      1487.7M  100%
      9601       3027       normal       80008.1GB       67725.7GB       12282.3GB    85% [#################   ]      1490.3M      1487.2M  100%
      9601       3028       normal       80008.1GB       66484.2GB       13523.9GB    83% [#################   ]      1490.3M      1487.2M  100%
      9601       3029       normal       80008.1GB       66775.8GB       13232.3GB    83% [#################   ]      1490.3M      1487.2M  100%
      9601       3030       normal       80008.1GB       67172.2GB       12835.8GB    84% [#################   ]      1490.3M      1487.2M  100%
      9601       3031       normal       80008.1GB       67583.0GB       12425.1GB    84% [#################   ]      1490.3M      1487.2M  100%
      9601       3032       normal       80008.1GB       67027.2GB       12980.9GB    84% [#################   ]      1490.3M      1487.2M  100%
      9601       3033       normal       80008.1GB       68068.1GB       11939.9GB    85% [#################   ]      1490.3M      1487.2M  100%
      9601       3034       normal       80008.1GB       66692.5GB       13315.6GB    83% [#################   ]      1490.3M      1487.2M  100%
      9602       3035       normal       80008.1GB       65461.4GB       14546.6GB    82% [################    ]      1490.3M      1487.2M  100%
      9602       3036       normal       80008.1GB       67415.6GB       12592.4GB    84% [#################   ]      1490.3M      1487.2M  100%
      9602       3037       normal       80008.1GB       66631.2GB       13376.9GB    83% [#################   ]      1490.3M      1487.2M  100%
      9602       3038    emergency       80008.1GB       67666.6GB       12341.5GB    85% [#################   ]      1490.3M      1487.7M  100%
      9602       3039       normal       80008.1GB       66443.1GB       13565.0GB    83% [#################   ]      1490.3M      1487.2M  100%
      9602       3040       normal       80008.1GB       66463.8GB       13544.2GB    83% [#################   ]      1490.3M      1487.2M  100%
      9602       3041       normal       80008.1GB       65779.8GB       14228.3GB    82% [################    ]      1490.3M      1487.2M  100%
      9701       3042       normal      119987.9GB       69566.5GB       50421.4GB    58% [############        ]      2235.0M      2225.1M  100%
      9701       3043       normal      119987.9GB       72282.5GB       47705.4GB    60% [############        ]      2235.0M      2225.1M  100%
      9701       3044       normal      119987.9GB       71015.8GB       48972.0GB    59% [############        ]      2235.0M      2225.1M  100%
      9701       3045       normal      119987.9GB       70551.7GB       49436.2GB    59% [############        ]      2235.0M      2225.1M  100%
      9701       3046       normal      119987.9GB       69538.0GB       50449.8GB    58% [############        ]      2235.0M      2225.1M  100%
      9701       3047       normal      119987.9GB       70626.6GB       49361.2GB    59% [############        ]      2235.0M      2225.1M  100%
      9701       3048       normal      119987.9GB       69932.6GB       50055.3GB    58% [############        ]      2235.0M      2225.1M  100%
      9701       3049       normal      119987.9GB       71488.6GB       48499.2GB    60% [############        ]      2235.0M      2225.1M  100%
      9701       3050       normal      119987.9GB       70340.5GB       49647.4GB    59% [############        ]      2235.0M      2225.1M  100%
      9701       3051       normal      119987.9GB       70832.5GB       49155.3GB    59% [############        ]      2235.0M      2225.1M  100%
      9701       3052       normal      119987.9GB       70039.6GB       49948.3GB    58% [############        ]      2235.0M      2225.1M  100%
      9701       3053       normal      119987.9GB       71313.3GB       48674.5GB    59% [############        ]      2235.0M      2225.1M  100%
      9701       3054       normal      119987.9GB       71179.6GB       48808.2GB    59% [############        ]      2235.0M      2225.1M  100%
      9701       3055       normal      119987.9GB       70700.6GB       49287.2GB    59% [############        ]      2235.0M      2225.1M  100%
      9702       3056       normal      119987.9GB       70656.3GB       49331.6GB    59% [############        ]      2235.0M      2225.1M  100%
      9702       3057       normal      119987.9GB       71425.6GB       48562.3GB    60% [############        ]      2235.0M      2225.1M  100%
      9702       3058       normal      119987.9GB       70542.2GB       49445.6GB    59% [############        ]      2235.0M      2225.1M  100%
      9702       3059       normal      119987.9GB       70758.5GB       49229.4GB    59% [############        ]      2235.0M      2225.1M  100%
      9702       3060       normal      119987.9GB       71263.5GB       48724.4GB    59% [############        ]      2235.0M      2225.1M  100%
      9702       3061       normal      119987.9GB       70712.2GB       49275.7GB    59% [############        ]      2235.0M      2225.1M  100%
      9702       3062       normal      119987.9GB       70814.1GB       49173.7GB    59% [############        ]      2235.0M      2225.1M  100%
      9702       3063       normal      119987.9GB       71426.7GB       48561.2GB    60% [############        ]      2235.0M      2225.1M  100%
      9702       3064       normal      119987.9GB       70687.0GB       49300.8GB    59% [############        ]      2235.0M      2225.1M  100%
      9702       3065       normal      119987.9GB       70469.6GB       49518.3GB    59% [############        ]      2235.0M      2225.1M  100%
      9702       3066       normal      119987.9GB       70784.3GB       49203.5GB    59% [############        ]      2235.0M      2225.1M  100%
      9702       3067       normal      119987.9GB       70514.2GB       49473.6GB    59% [############        ]      2235.0M      2225.1M  100%
      9702       3068       normal      119987.9GB       71143.3GB       48844.6GB    59% [############        ]      2235.0M      2225.1M  100%
      9702       3069       normal      119987.9GB       70301.4GB       49686.5GB    59% [############        ]      2235.0M      2225.1M  100%
      9801       3070       normal      119987.9GB       70162.4GB       49825.5GB    58% [############        ]      2235.0M      2225.1M  100%
      9801       3071       normal      119987.9GB       71569.4GB       48418.5GB    60% [############        ]      2235.0M      2228.0M  100%
      9801       3072       normal      119987.9GB       71592.0GB       48395.8GB    60% [############        ]      2235.0M      2225.1M  100%
      9801       3073       normal      119987.9GB       70329.0GB       49658.9GB    59% [############        ]      2235.0M      2225.1M  100%
      9801       3074       normal      119987.9GB       70957.1GB       49030.8GB    59% [############        ]      2235.0M      2225.1M  100%
      9801       3075       normal      119987.9GB       69796.8GB       50191.0GB    58% [############        ]      2235.0M      2225.1M  100%
      9801       3076       normal      119987.9GB       71328.0GB       48659.9GB    59% [############        ]      2235.0M      2225.1M  100%
      9801       3077       normal      119987.9GB       69925.5GB       50062.3GB    58% [############        ]      2235.0M      2225.1M  100%
      9801       3078       normal      119987.9GB       71101.2GB       48886.6GB    59% [############        ]      2235.0M      2225.1M  100%
      9801       3079       normal      119987.9GB       70829.6GB       49158.3GB    59% [############        ]      2235.0M      2225.1M  100%
      9801       3080       normal      119987.9GB       70331.2GB       49656.7GB    59% [############        ]      2235.0M      2227.9M  100%
      9801       3081       normal      119987.9GB       70090.4GB       49897.4GB    58% [############        ]      2235.0M      2225.1M  100%
      9801       3082       normal      119987.9GB       72090.7GB       47897.2GB    60% [############        ]      2235.0M      2225.1M  100%
      9801       3083       normal      119987.9GB       70566.3GB       49421.5GB    59% [############        ]      2235.0M      2225.1M  100%
      9802       3084       normal      119987.9GB       70139.4GB       49848.4GB    58% [############        ]      2235.0M      2225.1M  100%
      9802       3085       normal      119987.9GB       69517.0GB       50470.9GB    58% [############        ]      2235.0M      2225.1M  100%
      9802       3086       normal      119987.9GB       71315.1GB       48672.8GB    59% [############        ]      2235.0M      2225.1M  100%
      9802       3087       normal      119987.9GB       70990.8GB       48997.0GB    59% [############        ]      2235.0M      2225.1M  100%
      9802       3088       normal      119987.9GB       70191.8GB       49796.0GB    58% [############        ]      2235.0M      2225.1M  100%
      9802       3089       normal      119987.9GB       69887.4GB       50100.5GB    58% [############        ]      2235.0M      2225.1M  100%
      9802       3090       normal      119987.9GB       71714.7GB       48273.2GB    60% [############        ]      2235.0M      2225.1M  100%
      9802       3091       normal      119987.9GB       70674.0GB       49313.9GB    59% [############        ]      2235.0M      2225.1M  100%
      9802       3092       normal      119987.9GB       71048.1GB       48939.8GB    59% [############        ]      2235.0M      2225.1M  100%
      9802       3093       normal      119987.9GB       70616.7GB       49371.2GB    59% [############        ]      2235.0M      2225.1M  100%
      9802       3094       normal      119987.9GB       71696.2GB       48291.6GB    60% [############        ]      2235.0M      2225.1M  100%
      9802       3095       normal      119987.9GB       71475.0GB       48512.9GB    60% [############        ]      2235.0M      2225.1M  100%
      9802       3096       normal      119987.9GB       69776.4GB       50211.5GB    58% [############        ]      2235.0M      2225.1M  100%
      9802       3097       normal      119987.9GB       71002.7GB       48985.2GB    59% [############        ]      2235.0M      2225.1M  100%
      9502       3098       normal      119993.9GB       67162.7GB       52831.2GB    56% [###########         ]      2235.1M      2232.0M  100%
      9502       3099       normal      119993.9GB       67392.0GB       52601.9GB    56% [###########         ]      2235.1M      2232.0M  100%
      9502       3100       normal      119993.9GB       66447.8GB       53546.1GB    55% [###########         ]      2235.1M      2232.0M  100%
      9502       3101       normal      119993.9GB       65690.4GB       54303.5GB    55% [###########         ]      2235.1M      2232.0M  100%
      9502       3102       normal      119993.9GB       66074.7GB       53919.2GB    55% [###########         ]      2235.1M      2232.0M  100%
      9502       3103       normal      119993.9GB       67088.3GB       52905.6GB    56% [###########         ]      2235.1M      2232.0M  100%
      9502       3104       normal      119993.9GB       66653.8GB       53340.1GB    56% [###########         ]      2235.1M      2232.0M  100%
      9602       3105       normal      119993.9GB       65510.2GB       54483.7GB    55% [###########         ]      2235.1M      2232.0M  100%
      9602       3106       normal      119993.9GB       65322.0GB       54671.9GB    54% [###########         ]      2235.1M      2232.0M  100%
      9602       3107       normal      119993.9GB       66923.5GB       53070.4GB    56% [###########         ]      2235.1M      2232.0M  100%
      9602       3108       normal      119993.9GB       67485.2GB       52508.7GB    56% [###########         ]      2235.1M      2232.0M  100%
      9602       3109       normal      119993.9GB       67290.6GB       52703.3GB    56% [###########         ]      2235.1M      2232.0M  100%
      9602       3110       normal      119993.9GB       67184.6GB       52809.3GB    56% [###########         ]      2235.1M      2232.0M  100%
      9602       3111       normal      119993.9GB       66849.8GB       53144.0GB    56% [###########         ]      2235.1M      2232.0M  100%

Total size of /faststorage             11679565.5 GB
Total used space on /faststorage        7636403.1 GB
Total free space on /faststorage        4043162.4 GB

                                 Use%   65.38%

2020-09-21 07:51:22.626577974 +02:00
   SSSSSSSSSSSSSSS                                            iiii
 SS:::::::::::::::S                                          i::::i
S:::::SSSSSS::::::S                                           iiii
S:::::S     SSSSSSS
S:::::S            uuuuuu    uuuuuuvvvvvvv           vvvvvvviiiiiii
S:::::S            u::::u    u::::u v:::::v         v:::::v i:::::i
 S::::SSSS         u::::u    u::::u  v:::::v       v:::::v   i::::i
  SS::::::SSSSS    u::::u    u::::u   v:::::v     v:::::v    i::::i
    SSS::::::::SS  u::::u    u::::u    v:::::v   v:::::v     i::::i
       SSSSSS::::S u::::u    u::::u     v:::::v v:::::v      i::::i
            S:::::Su::::u    u::::u      v:::::v:::::v       i::::i
            S:::::Su:::::uuuu:::::u       v:::::::::v        i::::i
SSSSSSS     S:::::Su:::::::::::::::uu      v:::::::v        i::::::i
S::::::SSSSSS:::::S u:::::::::::::::u       v:::::v         i::::::i
S:::::::::::::::SS   uu::::::::uu:::u        v:::v          i::::::i
 SSSSSSSSSSSSSSS       uuuuuuuu  uuuu         vvv           iiiiiiii

                                        FreeBSD - The Power to Serve
[2020-09-21 07:50:02] s22n24 resumed
[2020-09-21 07:50:02] s21n82 resumed
[2020-09-21 07:50:02] s10n02 now marked for drain: health-check: df /faststorage hangs 
[2020-09-21 07:50:02] s03n18 resumed
[2020-09-21 07:48:01] s22n24 now marked for drain: health-check: df /faststorage hangs 
[2020-09-21 07:48:01] s21n82 now marked for drain: health-check: df /faststorage hangs 
[2020-09-21 07:48:01] s03n18 now marked for drain: health-check: df /faststorage hangs 
[2020-09-21 07:46:02] s21n33 resumed
[2020-09-21 07:46:02] s10n01 resumed
[2020-09-21 07:44:02] s21n42 resumed
[2020-09-21 07:44:02] s21n33 now marked for drain: health-check: df /faststorage hangs 
[2020-09-21 07:44:02] s21n14 resumed
[2020-09-21 07:44:02] s10n01 now marked for drain: health-check: df /faststorage hangs 
[2020-09-21 07:44:02] s05n23 resumed
[2020-09-21 07:42:02] s22n41 resumed
[2020-09-21 07:42:02] s22n24 resumed
[2020-09-21 07:42:02] s21n42 now marked for drain: health-check: df /faststorage hangs 
[2020-09-21 07:42:02] s21n14 now marked for drain: health-check: df /faststorage hangs 
[2020-09-21 07:42:02] s05n23 now marked for drain: health-check: df /faststorage hangs 
[2020-09-21 07:40:01] s22n41 now marked for drain: health-check: df /faststorage hangs 
[2020-09-21 07:40:01] s22n24 now marked for drain: health-check: df /faststorage hangs 
[2020-09-21 07:40:01] s21n64 resumed
[2020-09-21 07:40:01] s03n18 resumed
[2020-09-21 07:38:02] s21n64 now marked for drain: health-check: df /faststorage hangs 
[2020-09-21 07:38:02] s03n18 now marked for drain: health-check: df /faststorage hangs 
[2020-09-21 07:36:02] s22n44 resumed
[2020-09-21 07:36:02] s22n34 resumed
[2020-09-21 07:36:02] s21n61 resumed
[2020-09-21 07:34:02] s22n44 now marked for drain: health-check: df /faststorage hangs 
[2020-09-21 07:34:02] s22n34 now marked for drain: health-check: df /faststorage hangs 
[2020-09-21 07:34:02] s22n31 resumed
[2020-09-21 07:34:02] s21n82 resumed
[2020-09-21 07:34:02] s21n61 now marked for drain: health-check: df /faststorage hangs 
[2020-09-21 07:32:01] s22n34 resumed
[2020-09-21 07:32:01] s22n31 now marked for drain: health-check: df /faststorage hangs 
[2020-09-21 07:32:01] s21n82 now marked for drain: health-check: df /faststorage hangs 
[2020-09-21 07:30:02] s22n34 now marked for drain: health-check: df /faststorage hangs 
[2020-09-21 07:30:02] s05n22 resumed
[2020-09-21 07:28:02] s10n01 resumed
[2020-09-21 07:28:02] s05n22 now marked for drain: health-check: df /faststorage hangs 
[2020-09-21 07:26:02] s10n01 now marked for drain: health-check: df /faststorage hangs 
[2020-09-21 07:24:01] s21n64 resumed
[2020-09-21 07:22:02] s21n64 now marked for drain: health-check: df /faststorage hangs 
[2020-09-21 07:22:02] s21n61 resumed
[2020-09-21 07:20:02] s21n61 now marked for drain: health-check: df /faststorage hangs 
[2020-09-21 07:20:02] s05n22 resumed
[2020-09-21 07:18:02] s21n64 resumed
[2020-09-21 07:18:02] s05n22 now marked for drain: health-check: df /faststorage hangs 
[2020-09-21 07:16:01] s21n64 now marked for drain: health-check: df /faststorage hangs 
[2020-09-21 07:10:02] s22n44 resumed
[2020-09-21 07:10:02] s21n61 resumed
[2020-09-21 07:00:02]
Backup on    fs1  finished 0-04:39:19 ago
Backup on    fs2 initiated 4-13:15:00 ago
Backup on    fs3  finished 0-03:08:27 ago
Backup on    fs5 initiated 0-09:00:01 ago
Backup on    fs6  finished 0-01:36:51 ago
Backup on s95n01  finished 1-06:44:04 ago
[2020-09-21 07:50:02]
################
# RUNNING JOBS #
################
Count  USER         MIN_CPUS  MIN_MEMORY  TIMELIMIT    QOS
-----  ----         --------  ----------  ---------    ---
1      cactus       16        2G          12-12:00:00  basic
1      cactus       16        4G          30-00:00:00  basic
1      cactus       16        6G          30-00:00:00  basic
1      iastar       1         503G        3-00:00:00   qos1
1      mhra         1         128G        1-00:00:00   qos1
1      michaelk     16        16G         12:00:00     gold
1      rohde        1         30G         10:00:00     qos1
1      sla          64        128G        8:00:00      basic
2      cactus       16        2G          25-00:00:00  basic
2      liguangshuo  8         80G         4-00:00:00   bumped
2      michaelk     36        128G        12-00:00:00  gold
3      elivee       1         24G         4-00:00:00   qos1
3      mroussel     4         15G         20:00:00     qos1
4      vang         1         16G         2-00:00:00   gold
11     vang         2         8G          2-00:00:00   gold
46     mroussel     5         15G         20:00:00     qos1
66     vang         16        64G         2-00:00:00   gold
###############
# QUEUED JOBS #
###############
Count  USER   MIN_CPUS  MIN_MEMORY  TIMELIMIT  QOS
-----  ----   --------  ----------  ---------  ---
3223   lucie  1         110G        10:00:00   bumped
 
#################
# CPUs per User #
#################
NCPUs  USER         QOS
-----  ----         ---
1      iastar       qos1
1      mhra         qos1
1      rohde        qos1
3      elivee       qos1
16     liguangshuo  bumped
64     sla          basic
80     cactus       basic
88     michaelk     gold
242    mroussel     qos1
1082   vang         gold
########################
# QUEUED CPUs per User #
########################
NCPUs  USER   QOS
-----  ----   ---
3223   lucie  bumped