[2020-10-22 05:22:02]
         fs1  fs3  fs5  fs6
1  Gig   UP   UP   UP   UP 
10 Gig   n/a  UP   UP   UP 
IB       n/a  UP   UP   UP 
[2020-10-22 05:22:03]

 | 3-11|   OOOOOOOO | 3-21|   OOOOOOOO | 3-31|   ........ | 3-41|   OOOOOOO. | 3-51|   ........ | 3-61|   ........ | 3-71|   ........
 | 3-12|   OOOOOOOO | 3-22|   OOOOOOOO | 3-32|   ........ | 3-42|   OOOOOOOO | 3-52|   ........ | 3-62|   ........ | 3-73|   OOOOOOOO
 | 3-13|     DOWN   | 3-23|   OOOOOOOO | 3-33|   ........ | 3-43|   OOOOOOOO | 3-53|   ........ | 3-63|   ........ | 3-74|   OOOO....
 | 3-14|   OOOOOOOO | 3-24|   OOOOOOOO | 3-34|   ........ | 3-44|   OOOOOOOO | 3-54|   ........ | 3-64|   ........ | 3-75|   OOOOOOOO
 | 3-15|   OOOOOOOO | 3-26|   OOOOOOOO | 3-35|   ........ | 3-45|   ........ | 3-55|   ........ | 3-65|   ........ | 3-77|   OOOOOOOO
 | 3-16|   OOOOOOOO | 3-27|   OOOO.... | 3-36|   ........ | 3-46|   ........ | 3-56|   ........ | 3-67|   OOOOO... | 3-78|   ........
 | 3-17|   OOOOOOOO | 3-28|   ........ | 3-37|     DOWN   | 3-47|   ........ | 3-57|   ........ | 3-68|   ........                   
 | 3-18|   OOOOOOOO                    | 3-38|   OOOOOOO. | 3-48|   ........ | 3-58|   ........                                      


 | 4-11|   OOOOOOOO | 4-21|   OOOOOOO. | 4-31|   OOOOOOO. | 4-41|   OO...... | 4-51|   OOOOOOOO | 4-61|   OOOOO... | 4-71|   OOOOOOOO | 4-81|   ........
 | 4-12|   OOOOOOO. | 4-22|   OO...... | 4-32|   OOOOO... | 4-42|   OOOOOOO. | 4-52|   OOOOOOOO | 4-62|   OO...... | 4-72|   OOOOOOOO | 4-82|   ........
 | 4-13|   OOOOOOO. | 4-23|   OOOOOOO. | 4-33|   OOOOO... | 4-43|   OO...... | 4-53|   OOOOOOOO | 4-63|   OOOOOOOO | 4-73|   OOOOOOOO | 4-83|   OOOOO...
 | 4-14|   OOOOOOO. | 4-24|   OOOOO... | 4-34|     DOWN   | 4-44|   OOOOO... | 4-54|   OO...... | 4-64|   OOOOOOOO | 4-74|   OOOOOOOO                   


 | 5- 1|   OOOOOOO. | 5-11|   OOOOOOO. | 5-21|   OOOOOOOO
 | 5- 2|   OOOOOOO. | 5-12|   OOOOOO.. | 5-22|   OOOOOOOO
 | 5- 3|   OOOOOOO. | 5-13|   OOOOOOO. | 5-23|   OOOOOOOO
 | 5- 4|   OOOOOOO. | 5-14|   OOOOOO.. | 5-24|     DOWN  
 | 5- 5|   OOOOOOO. | 5-15|   OOOOOOO. | 5-25|     DOWN  
 | 5- 6|   OOOOOOO. | 5-16|   OOOOOOO. | 5-26|   OOOOOOOO
 | 5- 7|   OOOOOO.. | 5-17|   OOOOOOO. | 5-27|   OOOOOOOO
 | 5- 8|   OOOOOO.. | 5-18|     DOWN   | 5-28|   OOOOOOOO
 | 5- 9|   OOOOOOO. | 5-19|     DOWN   | 5-29|   OOOOOOOO
 | 5-10|   OOOOOO.. | 5-20|     DOWN                     


 | 9- 1|   ........
 | 9- 2|   ........
 | 9- 3|   ........


 |10- 1|   ........
 |10- 2|   ........


 |21-11|   OOO..... |21-21|   OOOOOOOO |21-31|   OOOOOOOO |21-41|   OOOOOOOO |21-51|   OOOOOOOO |21-61|   OOOOOOOO |21-71|   OOO..... |21-81|   OOOOOOOO |21-91|   OOOOOOOO
 |21-12|   OOOOOOOO |21-22|   OOOOOOOO |21-32|   OOOOOOOO |21-42|   OOOOOOOO |21-52|   OOOOOOOO |21-62|   OOOOOOOO |21-72|     DOWN   |21-82|   OOOOOOOO |21-92|   OOOOOOOO
 |21-13|   OOOOOOOO |21-23|   OOOOOOOO |21-33|   OOOOOOOO |21-43|   OOOOOOOO |21-53|   OOOOOOOO |21-63|   OOO..... |21-73|   OOOOOOOO |21-83|   OOOOOOOO |21-93|   OOOOOOOO
 |21-14|   OOOOOOOO |21-24|   OOOOOOO. |21-34|   OOOOOOOO |21-44|   OOOOOOOO |21-54|   OOOOOOOO |21-64|   OOOOOOO. |21-74|   OOOOOOO. |21-84|   OOOOOOOO |21-94|   OOOOOOOO


 |22-11|   OOOOOOOO |22-21|   OOOOOOOO |22-31|     DOWN   |22-41|   OOOOOOO. |22-51|   OOOOOOOO
 |22-12|   OOOOO... |22-22|   OOOOOO.. |22-32|   OOOOOOOO |22-42|   OOOOOOO. |22-52|   OOOOOO..
 |22-13|   OOOOOOOO |22-23|   OOOOOOO. |22-33|   OOOOOOO. |22-43|   OOOOOOOO |22-53| * OOOOO...
 |22-14|   OOOOOOOO |22-24|   OOOOOOOO |22-34|   OOOOOOOO |22-44|   OOOOOOOO                   


 |91- 1|   OOOOOOOO
 |91- 2|   OOOOOOOO
 |91- 3|   OOOOOOOO
 |91- 4|   OOOOOOOO
 |91- 5|   OOOOOOOO
 |91- 6|   OOOOOOOO


Totals:
Total cores: 6404 
Total memory: 56.67 TiB 

Down:
Down cores: 412 (  6.43%) 
Down memory: 3.54 TiB (  6.24%) 

Used:
Used cores: 4570 ( 71.36%) 
Used memory: 32.23 TiB ( 56.87%) 

Utilization excluding down
CPU utilization:  76.27% (global)  64.00% (average per node)
Memory utilization:  60.66% (global)  54.31% (average per node)

Average per node utilization (excluding down):  72.39% 
[2020-10-22 05:22:03]
s03n13 health-check: s03n13 25-Aug-2020 10:01 Lost part of memory 
s03n37 health-check: Lost part of memory 
s04n34 Not responding 
s05n18 pon/pof doesn't work 
s05n19 pon/pof doesn't work 
s05n20 pon/pof doesn't work 
s05n24 pon/pof doesn't work 
s05n25 pon/pof doesn't work 
s21n72 Not responding 
s22n31 Not responding 
s22n53 firmware update 

[2020-10-22 05:20:15]
[store03 s94n01] Controller 0
[store03 s94n01] Virtual Drive  0 - RAID1 - UU - Optimal
[store03 s94n01] 

[store03 s94n02] Controller 0
[store03 s94n02] Virtual Drive  0 - RAID1 - UU - Optimal
[store03 s94n02] 

[store03 s95n01] Controller 0
[store03 s95n01] Virtual Drive  0 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n01] Virtual Drive  1 - RAID6 - UUU_UUUUUUUU - Partially Degraded - Rebuild: 69% ETA: 19 Hours 39 Minutes 
[store03 s95n01] Virtual Drive  2 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n01] Virtual Drive  3 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n01] Virtual Drive  4 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n01] Virtual Drive  5 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n01] Virtual Drive  6 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n01] Controller 1
[store03 s95n01] Virtual Drive  0 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n01] Virtual Drive  1 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n01] Virtual Drive  2 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n01] Virtual Drive  3 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n01] Virtual Drive  4 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n01] Virtual Drive  5 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n01] Virtual Drive  6 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n01] 

[store03 s95n02] Controller 0
[store03 s95n02] Virtual Drive  0 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n02] Virtual Drive  1 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n02] Virtual Drive  2 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n02] Virtual Drive  3 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n02] Virtual Drive  4 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n02] Virtual Drive  5 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n02] Virtual Drive  6 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n02] Virtual Drive  7 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n02] Virtual Drive  8 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n02] Virtual Drive  9 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n02] Virtual Drive 10 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n02] Virtual Drive 11 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n02] Virtual Drive 12 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n02] Virtual Drive 13 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n02] 

[store03 s96n01] Controller 0
[store03 s96n01] Virtual Drive  7 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n01] Virtual Drive  8 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n01] Virtual Drive  9 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n01] Virtual Drive 10 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n01] Virtual Drive 11 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n01] Virtual Drive 12 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n01] Virtual Drive 13 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n01] Controller 1
[store03 s96n01] Virtual Drive  0 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n01] Virtual Drive  1 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n01] Virtual Drive  2 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n01] Virtual Drive  3 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n01] Virtual Drive  4 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n01] Virtual Drive  5 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n01] Virtual Drive  6 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n01] 

[store03 s96n02] Controller 0
[store03 s96n02] Virtual Drive  0 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n02] Virtual Drive  1 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n02] Virtual Drive  2 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n02] Virtual Drive  3 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n02] Virtual Drive  4 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n02] Virtual Drive  5 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n02] Virtual Drive  6 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n02] Virtual Drive  7 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n02] Virtual Drive  8 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n02] Virtual Drive  9 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n02] Virtual Drive 10 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n02] Virtual Drive 11 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n02] Virtual Drive 12 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n02] Virtual Drive 13 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n02] 

[store03 s97n01] Controller 0
[store03 s97n01] Virtual Drive  0 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n01] Virtual Drive  1 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n01] Virtual Drive  2 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n01] Virtual Drive  3 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n01] Virtual Drive  4 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n01] Virtual Drive  5 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n01] Virtual Drive  6 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n01] Controller 1
[store03 s97n01] Virtual Drive  0 - RAID1 - UU - Optimal
[store03 s97n01] Controller 2
[store03 s97n01] Virtual Drive  0 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n01] Virtual Drive  1 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n01] Virtual Drive  2 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n01] Virtual Drive  3 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n01] Virtual Drive  4 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n01] Virtual Drive  5 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n01] Virtual Drive  6 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n01] 

[store03 s97n02] Controller 0
[store03 s97n02] Virtual Drive  0 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n02] Virtual Drive  1 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n02] Virtual Drive  2 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n02] Virtual Drive  3 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n02] Virtual Drive  4 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n02] Virtual Drive  5 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n02] Virtual Drive  6 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n02] Controller 1
[store03 s97n02] Virtual Drive  0 - RAID1 - UU - Optimal
[store03 s97n02] Controller 2
[store03 s97n02] Virtual Drive  0 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n02] Virtual Drive  1 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n02] Virtual Drive  2 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n02] Virtual Drive  3 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n02] Virtual Drive  4 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n02] Virtual Drive  5 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n02] Virtual Drive  6 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n02] 

[store03 s98n01] Controller 0
[store03 s98n01] Virtual Drive  0 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n01] Virtual Drive  1 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n01] Virtual Drive  2 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n01] Virtual Drive  3 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n01] Virtual Drive  4 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n01] Virtual Drive  5 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n01] Virtual Drive  6 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n01] Controller 1
[store03 s98n01] Virtual Drive  0 - RAID1 - UU - Optimal
[store03 s98n01] Controller 2
[store03 s98n01] Virtual Drive  0 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n01] Virtual Drive  1 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n01] Virtual Drive  2 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n01] Virtual Drive  3 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n01] Virtual Drive  4 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n01] Virtual Drive  5 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n01] Virtual Drive  6 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n01] 

[store03 s98n02] Controller 0
[store03 s98n02] Virtual Drive  0 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n02] Virtual Drive  1 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n02] Virtual Drive  2 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n02] Virtual Drive  3 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n02] Virtual Drive  4 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n02] Virtual Drive  5 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n02] Virtual Drive  6 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n02] Controller 1
[store03 s98n02] Virtual Drive  0 - RAID1 - UU - Optimal
[store03 s98n02] Controller 2
[store03 s98n02] Virtual Drive  0 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n02] Virtual Drive  1 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n02] Virtual Drive  2 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n02] Virtual Drive  3 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n02] Virtual Drive  4 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n02] Virtual Drive  5 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n02] Virtual Drive  6 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n02] 

[2020-10-22 05:15:05]
    NodeID   TargetID         Pool           Total            Used            Free   Use% Usage Bar                    ITotal        IFree     %
    ======   ========         ====           =====            ====            ====   ==== =========                    ======        =====     =
      9501       3001    emergency       80008.1GB       64248.6GB       15759.5GB    80% [################    ]      1490.3M      1487.7M  100%
      9501       3002       normal       80008.1GB       67531.1GB       12476.9GB    84% [#################   ]      1490.3M      1487.1M  100%
      9501       3003       normal       80008.1GB       68266.5GB       11741.6GB    85% [#################   ]      1490.3M      1487.1M  100%
      9501       3004       normal       80008.1GB       69026.2GB       10981.8GB    86% [#################   ]      1490.3M      1487.1M  100%
      9501       3005       normal       80008.1GB       66290.0GB       13718.0GB    83% [#################   ]      1490.3M      1487.1M  100%
      9501       3006       normal       80008.1GB       69089.3GB       10918.8GB    86% [#################   ]      1490.3M      1487.1M  100%
      9501       3007       normal       80008.1GB       70256.8GB        9751.3GB    88% [##################  ]      1490.3M      1487.2M  100%
      9501       3008       normal       80008.1GB       69014.5GB       10993.6GB    86% [#################   ]      1490.3M      1487.1M  100%
      9501       3009       normal       80008.1GB       69074.3GB       10933.8GB    86% [#################   ]      1490.3M      1487.1M  100%
      9501       3010       normal       80008.1GB       67510.8GB       12497.3GB    84% [#################   ]      1490.3M      1487.1M  100%
      9501       3011       normal       80008.1GB       65055.8GB       14952.3GB    81% [################    ]      1490.3M      1487.1M  100%
      9501       3012       normal       80008.1GB       67926.3GB       12081.7GB    85% [#################   ]      1490.3M      1487.1M  100%
      9501       3013       normal       80008.1GB       66144.2GB       13863.9GB    83% [#################   ]      1490.3M      1487.0M  100%
      9502       3014       normal       80008.1GB       65573.7GB       14434.4GB    82% [################    ]      1490.3M      1487.0M  100%
      9502       3015       normal       80008.1GB       67327.8GB       12680.3GB    84% [#################   ]      1490.3M      1487.0M  100%
      9502       3016       normal       80008.1GB       70873.5GB        9134.6GB    89% [##################  ]      1490.3M      1489.4M  100%
      9502       3017       normal       80008.1GB       67404.4GB       12603.7GB    84% [#################   ]      1490.3M      1487.0M  100%
      9502       3018       normal       80008.1GB       68387.7GB       11620.4GB    85% [#################   ]      1490.3M      1487.0M  100%
      9502       3019       normal       80008.1GB       66537.3GB       13470.8GB    83% [#################   ]      1490.3M      1487.0M  100%
      9502       3020       normal       80008.1GB       67645.8GB       12362.2GB    85% [#################   ]      1490.3M      1487.0M  100%
      9601       3021       normal       80008.1GB       67228.2GB       12779.9GB    84% [#################   ]      1490.3M      1487.1M  100%
      9601       3022       normal       80008.1GB       66685.2GB       13322.9GB    83% [#################   ]      1490.3M      1487.1M  100%
      9601       3023       normal       80008.1GB       67317.2GB       12690.8GB    84% [#################   ]      1490.3M      1487.1M  100%
      9601       3024       normal       80008.1GB       68722.0GB       11286.0GB    86% [#################   ]      1490.3M      1487.1M  100%
      9601       3025       normal       80008.1GB       67113.7GB       12894.3GB    84% [#################   ]      1490.3M      1487.1M  100%
      9601       3026    emergency       80008.1GB       63564.0GB       16444.1GB    79% [################    ]      1490.3M      1487.7M  100%
      9601       3027       normal       80008.1GB       70405.1GB        9603.0GB    88% [##################  ]      1490.3M      1487.1M  100%
      9601       3028       normal       80008.1GB       67156.9GB       12851.2GB    84% [#################   ]      1490.3M      1487.1M  100%
      9601       3029       normal       80008.1GB       67182.1GB       12825.9GB    84% [#################   ]      1490.3M      1487.1M  100%
      9601       3030       normal       80008.1GB       67747.2GB       12260.9GB    85% [#################   ]      1490.3M      1487.1M  100%
      9601       3031       normal       80008.1GB       69543.6GB       10464.5GB    87% [#################   ]      1490.3M      1487.1M  100%
      9601       3032       normal       80008.1GB       67598.0GB       12410.1GB    84% [#################   ]      1490.3M      1487.1M  100%
      9601       3033       normal       80008.1GB       68732.7GB       11275.4GB    86% [#################   ]      1490.3M      1487.1M  100%
      9601       3034       normal       80008.1GB       67259.2GB       12748.9GB    84% [#################   ]      1490.3M      1487.1M  100%
      9602       3035       normal       80008.1GB       65931.0GB       14077.0GB    82% [################    ]      1490.3M      1487.1M  100%
      9602       3036       normal       80008.1GB       68047.4GB       11960.6GB    85% [#################   ]      1490.3M      1487.1M  100%
      9602       3037       normal       80008.1GB       67103.4GB       12904.6GB    84% [#################   ]      1490.3M      1487.1M  100%
      9602       3038       normal       80008.1GB       67794.5GB       12213.6GB    85% [#################   ]      1490.3M      1487.7M  100%
      9602       3039       normal       80008.1GB       67845.7GB       12162.3GB    85% [#################   ]      1490.3M      1487.1M  100%
      9602       3040       normal       80008.1GB       66865.5GB       13142.6GB    84% [#################   ]      1490.3M      1487.2M  100%
      9602       3041       normal       80008.1GB       66684.0GB       13324.1GB    83% [#################   ]      1490.3M      1487.1M  100%
      9701       3042       normal      119987.9GB       69238.1GB       50749.7GB    58% [############        ]      2235.0M      2225.0M  100%
      9701       3043       normal      119987.9GB       71555.5GB       48432.3GB    60% [############        ]      2235.0M      2225.0M  100%
      9701       3044       normal      119987.9GB       70512.9GB       49474.9GB    59% [############        ]      2235.0M      2225.0M  100%
      9701       3045       normal      119987.9GB       70283.6GB       49704.3GB    59% [############        ]      2235.0M      2225.0M  100%
      9701       3046       normal      119987.9GB       69458.8GB       50529.0GB    58% [############        ]      2235.0M      2225.0M  100%
      9701       3047       normal      119987.9GB       70165.1GB       49822.8GB    58% [############        ]      2235.0M      2225.0M  100%
      9701       3048       normal      119987.9GB       69482.6GB       50505.2GB    58% [############        ]      2235.0M      2225.0M  100%
      9701       3049       normal      119987.9GB       71307.0GB       48680.9GB    59% [############        ]      2235.0M      2225.0M  100%
      9701       3050       normal      119987.9GB       69480.3GB       50507.6GB    58% [############        ]      2235.0M      2225.0M  100%
      9701       3051       normal      119987.9GB       70179.2GB       49808.6GB    58% [############        ]      2235.0M      2225.0M  100%
      9701       3052       normal      119987.9GB       69243.6GB       50744.2GB    58% [############        ]      2235.0M      2225.0M  100%
      9701       3053       normal      119987.9GB       70852.7GB       49135.2GB    59% [############        ]      2235.0M      2225.0M  100%
      9701       3054       normal      119987.9GB       70574.6GB       49413.2GB    59% [############        ]      2235.0M      2225.0M  100%
      9701       3055       normal      119987.9GB       69983.5GB       50004.4GB    58% [############        ]      2235.0M      2225.0M  100%
      9702       3056       normal      119987.9GB       70275.8GB       49712.0GB    59% [############        ]      2235.0M      2225.0M  100%
      9702       3057       normal      119987.9GB       71201.5GB       48786.4GB    59% [############        ]      2235.0M      2225.0M  100%
      9702       3058       normal      119987.9GB       70295.6GB       49692.2GB    59% [############        ]      2235.0M      2225.0M  100%
      9702       3059       normal      119987.9GB       70219.7GB       49768.2GB    59% [############        ]      2235.0M      2225.0M  100%
      9702       3060       normal      119987.9GB       70828.4GB       49159.5GB    59% [############        ]      2235.0M      2225.0M  100%
      9702       3061       normal      119987.9GB       69800.0GB       50187.9GB    58% [############        ]      2235.0M      2225.0M  100%
      9702       3062       normal      119987.9GB       69936.9GB       50051.0GB    58% [############        ]      2235.0M      2225.0M  100%
      9702       3063       normal      119987.9GB       70871.8GB       49116.0GB    59% [############        ]      2235.0M      2225.0M  100%
      9702       3064       normal      119987.9GB       70145.8GB       49842.1GB    58% [############        ]      2235.0M      2225.0M  100%
      9702       3065       normal      119987.9GB       69633.0GB       50354.9GB    58% [############        ]      2235.0M      2225.0M  100%
      9702       3066       normal      119987.9GB       70523.0GB       49464.8GB    59% [############        ]      2235.0M      2225.0M  100%
      9702       3067       normal      119987.9GB       69940.9GB       50046.9GB    58% [############        ]      2235.0M      2225.0M  100%
      9702       3068       normal      119987.9GB       72589.3GB       47398.6GB    60% [############        ]      2235.0M      2225.0M  100%
      9702       3069       normal      119987.9GB       69503.5GB       50484.3GB    58% [############        ]      2235.0M      2225.0M  100%
      9801       3070       normal      119987.9GB       69608.7GB       50379.2GB    58% [############        ]      2235.0M      2225.1M  100%
      9801       3071       normal      119987.9GB       70867.2GB       49120.7GB    59% [############        ]      2235.0M      2227.9M  100%
      9801       3072       normal      119987.9GB       72449.5GB       47538.4GB    60% [############        ]      2235.0M      2225.0M  100%
      9801       3073       normal      119987.9GB       69078.5GB       50909.4GB    58% [############        ]      2235.0M      2225.0M  100%
      9801       3074       normal      119987.9GB       71129.2GB       48858.7GB    59% [############        ]      2235.0M      2225.0M  100%
      9801       3075       normal      119987.9GB       69045.3GB       50942.5GB    58% [############        ]      2235.0M      2225.0M  100%
      9801       3076       normal      119987.9GB       72577.5GB       47410.3GB    60% [############        ]      2235.0M      2225.0M  100%
      9801       3077       normal      119987.9GB       69399.1GB       50588.7GB    58% [############        ]      2235.0M      2225.0M  100%
      9801       3078       normal      119987.9GB       70738.5GB       49249.3GB    59% [############        ]      2235.0M      2225.0M  100%
      9801       3079       normal      119987.9GB       71803.7GB       48184.2GB    60% [############        ]      2235.0M      2225.0M  100%
      9801       3080       normal      119987.9GB       69653.7GB       50334.1GB    58% [############        ]      2235.0M      2227.8M  100%
      9801       3081       normal      119987.9GB       69679.4GB       50308.5GB    58% [############        ]      2235.0M      2225.0M  100%
      9801       3082       normal      119987.9GB       71449.1GB       48538.8GB    60% [############        ]      2235.0M      2225.0M  100%
      9801       3083       normal      119987.9GB       69855.9GB       50132.0GB    58% [############        ]      2235.0M      2225.0M  100%
      9802       3084       normal      119987.9GB       69850.5GB       50137.4GB    58% [############        ]      2235.0M      2225.0M  100%
      9802       3085       normal      119987.9GB       68851.3GB       51136.5GB    57% [###########         ]      2235.0M      2225.0M  100%
      9802       3086       normal      119987.9GB       70768.4GB       49219.5GB    59% [############        ]      2235.0M      2225.0M  100%
      9802       3087       normal      119987.9GB       70421.9GB       49566.0GB    59% [############        ]      2235.0M      2225.0M  100%
      9802       3088       normal      119987.9GB       69500.6GB       50487.2GB    58% [############        ]      2235.0M      2225.0M  100%
      9802       3089       normal      119987.9GB       68986.5GB       51001.3GB    57% [###########         ]      2235.0M      2225.0M  100%
      9802       3090       normal      119987.9GB       71118.9GB       48869.0GB    59% [############        ]      2235.0M      2225.0M  100%
      9802       3091       normal      119987.9GB       70391.8GB       49596.1GB    59% [############        ]      2235.0M      2225.0M  100%
      9802       3092       normal      119987.9GB       70201.3GB       49786.6GB    59% [############        ]      2235.0M      2225.0M  100%
      9802       3093       normal      119987.9GB       70727.9GB       49260.0GB    59% [############        ]      2235.0M      2225.0M  100%
      9802       3094       normal      119987.9GB       70930.4GB       49057.4GB    59% [############        ]      2235.0M      2225.0M  100%
      9802       3095       normal      119987.9GB       70803.9GB       49184.0GB    59% [############        ]      2235.0M      2225.0M  100%
      9802       3096       normal      119987.9GB       69134.4GB       50853.5GB    58% [############        ]      2235.0M      2225.0M  100%
      9802       3097       normal      119987.9GB       70459.1GB       49528.7GB    59% [############        ]      2235.0M      2225.0M  100%
      9502       3098       normal      119993.9GB       67542.8GB       52451.1GB    56% [###########         ]      2235.1M      2231.9M  100%
      9502       3099       normal      119993.9GB       67762.7GB       52231.2GB    56% [###########         ]      2235.1M      2231.9M  100%
      9502       3100       normal      119993.9GB       67422.7GB       52571.2GB    56% [###########         ]      2235.1M      2231.9M  100%
      9502       3101       normal      119993.9GB       67027.3GB       52966.6GB    56% [###########         ]      2235.1M      2231.9M  100%
      9502       3102       normal      119993.9GB       67078.6GB       52915.3GB    56% [###########         ]      2235.1M      2231.9M  100%
      9502       3103       normal      119993.9GB       67871.4GB       52122.5GB    57% [###########         ]      2235.1M      2231.9M  100%
      9502       3104       normal      119993.9GB       66914.3GB       53079.6GB    56% [###########         ]      2235.1M      2231.9M  100%
      9602       3105       normal      119993.9GB       66275.4GB       53718.5GB    55% [###########         ]      2235.1M      2231.9M  100%
      9602       3106       normal      119993.9GB       65129.0GB       54864.9GB    54% [###########         ]      2235.1M      2232.0M  100%
      9602       3107       normal      119993.9GB       67223.9GB       52770.0GB    56% [###########         ]      2235.1M      2231.9M  100%
      9602       3108       normal      119993.9GB       68228.6GB       51765.3GB    57% [###########         ]      2235.1M      2231.9M  100%
      9602       3109       normal      119993.9GB       69821.6GB       50172.3GB    58% [############        ]      2235.1M      2231.9M  100%
      9602       3110       normal      119993.9GB       67577.8GB       52416.1GB    56% [###########         ]      2235.1M      2231.9M  100%
      9602       3111       normal      119993.9GB       67690.2GB       52303.7GB    56% [###########         ]      2235.1M      2231.9M  100%

Total size of /faststorage             11679565.5 GB
Total used space on /faststorage        7650842.2 GB
Total free space on /faststorage        4028723.3 GB

                                 Use%   65.51%

2020-10-22 05:23:33.609481771 +02:00
   SSSSSSSSSSSSSSS                                            iiii
 SS:::::::::::::::S                                          i::::i
S:::::SSSSSS::::::S                                           iiii
S:::::S     SSSSSSS
S:::::S            uuuuuu    uuuuuuvvvvvvv           vvvvvvviiiiiii
S:::::S            u::::u    u::::u v:::::v         v:::::v i:::::i
 S::::SSSS         u::::u    u::::u  v:::::v       v:::::v   i::::i
  SS::::::SSSSS    u::::u    u::::u   v:::::v     v:::::v    i::::i
    SSS::::::::SS  u::::u    u::::u    v:::::v   v:::::v     i::::i
       SSSSSS::::S u::::u    u::::u     v:::::v v:::::v      i::::i
            S:::::Su::::u    u::::u      v:::::v:::::v       i::::i
            S:::::Su:::::uuuu:::::u       v:::::::::v        i::::i
SSSSSSS     S:::::Su:::::::::::::::uu      v:::::::v        i::::::i
S::::::SSSSSS:::::S u:::::::::::::::u       v:::::v         i::::::i
S:::::::::::::::SS   uu::::::::uu:::u        v:::v          i::::::i
 SSSSSSSSSSSSSSS       uuuuuuuu  uuuu         vvv           iiiiiiii

                                        FreeBSD - The Power to Serve
[2020-10-21 20:12:03] s22n14 resumed
[2020-10-21 19:42:02] s04n54 resumed
[2020-10-21 19:40:03] s04n54 now marked for drain: health-check: can't ping gfs3 
[2020-10-21 19:26:07] s03n74 resumed
[2020-10-21 19:24:04] s03n74 now marked for drain: health-check: can't ping gfs5 
[2020-10-21 18:08:02] s04n73 resumed
[2020-10-21 18:06:03] s04n73 now marked for drain: health-check: can't ping gfs5 
[2020-10-21 15:24:02] s22n14 now marked for drain: health-check: over 50% swap space used 
[2020-10-20 16:54:02] s04n63 resumed
[2020-10-20 16:52:02] s04n63 now marked for drain: health-check: can't ping gfs3 
[2020-10-20 13:16:02] s03n53 resumed
[2020-10-20 13:14:01] s03n53 now marked for drain: health-check: can't ping gfs5 
[2020-10-20 12:50:01] s04n63 resumed
[2020-10-20 12:48:02] s04n63 now marked for drain: health-check: can't ping gfs3 
[2020-10-20 09:02:02] s03n42 resumed
[2020-10-20 09:00:01] s03n42 now marked for drain: health-check: can't ping gfs3 
[2020-10-20 05:24:02] s22n31 now marked for drain: Not responding 
[2020-10-19 17:28:02] s21n13 resumed
[2020-10-19 17:26:02] s21n13 now marked for drain: health-check: can't ping gfs6 
[2020-10-19 17:24:01] s21n72 now marked for drain: Not responding 
[2020-10-19 16:40:01] s22n22 resumed
[2020-10-19 16:38:03] s22n23 resumed
[2020-10-19 16:30:02] s22n21 resumed
[2020-10-19 16:28:02] s22n24 resumed
[2020-10-19 16:20:02] s22n24 now marked for drain: health-check: over 50% swap space used 
[2020-10-19 16:20:02] s22n23 now marked for drain: health-check: over 50% swap space used 
[2020-10-19 16:20:02] s22n21 now marked for drain: health-check: over 50% swap space used 
[2020-10-19 16:18:02] s22n22 now marked for drain: health-check: over 50% swap space used 
[2020-10-19 16:18:02] s03n27 resumed
[2020-10-19 16:16:02] s03n27 now marked for drain: health-check: can't ping gfs3 
[2020-10-19 09:44:02] s03n73 resumed
[2020-10-18 11:16:02] s03n37 now marked for drain: needs reboot 
[2020-10-18 11:14:02] s03n73 now marked for drain: needs reboot 
[2020-10-18 07:00:02] s91n05 resumed
[2020-10-18 07:00:02] s91n03 resumed
[2020-10-18 07:00:02] s21n81 resumed
[2020-10-18 07:00:02] s21n73 resumed
[2020-10-18 07:00:02] s21n54 resumed
[2020-10-18 07:00:02] s05n06 resumed
[2020-10-18 07:00:02] s04n43 resumed
[2020-10-18 07:00:02] s03n47 resumed
[2020-10-18 07:00:02] s03n34 resumed
[2020-10-18 06:58:02] s91n05 now marked for drain: health-check: /faststorage not available 
[2020-10-18 06:58:02] s91n03 now marked for drain: health-check: /faststorage not available 
[2020-10-18 06:58:02] s21n81 now marked for drain: health-check: /faststorage not available 
[2020-10-18 06:58:02] s21n73 now marked for drain: health-check: /faststorage not available 
[2020-10-18 06:58:02] s21n54 now marked for drain: health-check: /faststorage not available 
[2020-10-18 06:58:02] s05n06 now marked for drain: health-check: /faststorage not available 
[2020-10-18 06:58:02] s04n43 now marked for drain: health-check: /faststorage not available 
[2020-10-18 06:58:02] s03n47 now marked for drain: health-check: /faststorage not available 
[2020-10-18 06:58:02] s03n34 now marked for drain: health-check: /faststorage not available 
[2020-10-22 05:00:02]
Backup on    fs1 initiated 0-06:30:00 ago
Backup on    fs2 initiated 14-11:15:00 ago
Backup on    fs3 initiated 0-03:00:00 ago
Backup on    fs5 initiated 0-07:00:01 ago
Backup on    fs6 initiated 0-03:00:01 ago
Backup on s95n01  finished 0-05:50:05 ago
[2020-10-22 05:22:05]
################
# RUNNING JOBS #
################
Count  USER         MIN_CPUS  MIN_MEMORY  TIMELIMIT    QOS
-----  ----         --------  ----------  ---------    ---
1      cactus       16        10G         30-00:00:00  basic
1      cactus       16        4G          44-00:00:00  basic
1      cactus       16        6G          44-00:00:00  basic
1      clara        16        128G        1-00:00:00   qos1
1      kmt          1         40G         14-00:00:00  qos1
1      liguangshuo  8         40G         4-00:00:00   basic
1      michaelk     1         16G         12:00:00     qos1
1      simond       1         20G         20:00:00     basic
1      vang         1         16G         1-00:00:00   gold
1      vang         1         4G          12:00:00     gold
1      vang         1         64G         12:00:00     gold
1      vang         8         4G          1-00:00:00   gold
2      astamr       16        32G         1-18:00:00   bumped
2      cactus       16        20G         30-00:00:00  basic
2      cactus       16        6G          30-00:00:00  basic
2      michaelk     1         32G         12:00:00     qos1
3      il           15        60G         6:00:00      qos1
3      vang         1         8G          12:00:00     gold
5      fprive       30        250G        12:00:00     qos1
5      il           1         60G         12:00:00     qos1
5      il           1         60G         3-00:00:00   qos1
5      sla          64        503G        12:00:00     normal
10     vang         8         128G        2-00:00:00   gold
23     clara        32        250G        2-00:00:00   qos1
62     besen        1         80G         1-08:00:00   qos1
90     vang         24        128G        2-00:00:00   gold
205    vang         4         16G         12:00:00     gold
###############
# QUEUED JOBS #
###############
Count  USER   MIN_CPUS  MIN_MEMORY  TIMELIMIT   QOS
-----  ----   --------  ----------  ---------   ---
1      il     15        60G         6:00:00     qos1
20     vang   24        128G        2-00:00:00  gold
40     palle  8         500G        1-00:00:00  qos1
67     clara  32        250G        2-00:00:00  qos1
762    sla    64        503G        12:00:00    normal
 
#################
# CPUs per User #
#################
NCPUs  USER         QOS
-----  ----         ---
1      kmt          qos1
1      simond       basic
3      michaelk     qos1
8      liguangshuo  basic
32     astamr       bumped
55     il           qos1
62     besen        qos1
112    cactus       basic
150    fprive       qos1
320    sla          normal
752    clara        qos1
3074   vang         gold
########################
# QUEUED CPUs per User #
########################
NCPUs  USER   QOS
-----  ----   ---
15     il     qos1
320    palle  qos1
480    vang   gold
2144   clara  qos1
48768  sla    normal