[2019-12-05 15:42:02]
         fs1  fs3  fs4  fs5  fs6
1  Gig   UP   UP   UP   UP   UP 
10 Gig   n/a  UP   UP   UP   UP 
IB       n/a  UP   UP   UP   UP 
[2019-12-05 15:42:03]

 | 3-11|   OOOOOOOO | 3-21|   OOOO.... | 3-31|   OOO..... | 3-41|   OOOOOOOO | 3-51|   OOOOOO.. | 3-61|   OOOOOO.. | 3-71|   OOOOOOO.
 | 3-12|   OOOO.... | 3-22|   OOOOOOOO | 3-32|   OOOOOOOO | 3-42|   OOOOOOO. | 3-52|   OO...... | 3-62|   OOOOOOO. | 3-72|   OOOOOO..
 | 3-13|   OOOOOOOO | 3-23|   OOOO.... | 3-33|   OOOOOOOO | 3-43|   OOOOOOOO | 3-53|   OOOOOOO. | 3-63|   OOOOOOO. | 3-73|   OOOOOOO.
 | 3-14|   OOOOOOOO | 3-24|   OOOOOOOO | 3-34|   OOOOOOOO | 3-44|   OOOOOOOO | 3-54|   OOOOOOO. | 3-64|   OOOOOOOO | 3-74|   OOOOOO..
 | 3-15|   OOOOOOOO | 3-25|   OOOOOOOO | 3-35|   OOOOOOOO | 3-45|   OOOOOOOO | 3-55|   OOOOOO.. | 3-65|   OOOO.... | 3-75|   OOOO....
 | 3-16|   OOOOOOOO | 3-26|   OOOOOOOO | 3-36|   OOOOOOOO | 3-46|   OOOOOOO. | 3-56|   OOOOOO.. | 3-66|   OOOOOOO. | 3-76|   O.......
 | 3-17|   OOOO.... | 3-27|   OOOOOOOO | 3-37|   OOOOOOOO | 3-47|   OOOOOOO. | 3-57|   OOOOOOO. | 3-67|   OOOOOOO. | 3-77|   OOOOOO..
 | 3-18|   OOOOOOOO | 3-28|   OOOOOOOO | 3-38|   OOOOOOOO | 3-48|   OOOOOOO. | 3-58|   OOOOOOO. | 3-68|   OOOOOO.. | 3-78|   OO......


 | 4-11|   OOOOOOOO | 4-21|   OOOOOOO. | 4-31|   OOOOOOOO | 4-41|   OOOOOOO. | 4-51|   OOOOOOO. | 4-61|   OOOOOOO. | 4-71|   OOOOOOO. | 4-81|   OOOOOOO. | 4-91|   OOOOOO..
 | 4-12|   OOOOOOOO | 4-22|   OOOOOOOO | 4-32|   OOOOOOOO | 4-42|   OOOOOO.. | 4-52|   OOOOOO.. | 4-62|   OOOOOOO. | 4-72|   OOOOOO.. | 4-82|   OOOOO... | 4-92|   OOOOOO..
 | 4-13|   OOOOOO.. | 4-23|   OOOOOOO. | 4-33|   OOOOOOOO | 4-43|   OOOOOO.. | 4-53|   OOOOOOO. | 4-63|   OOOOOO.. | 4-73|   OOOOOOO. | 4-83|   OOOOOOOO
 | 4-14|   OOOOOOOO | 4-24|   OOOOOO.. | 4-34|   OOOOOOO. | 4-44|   OOOOOOOO | 4-54|   OOOOOOOO | 4-64|   OOOOOO.. | 4-74|   OOO..... | 4-84|   OOOOOOOO


 | 5- 1|   OOOOOOOO | 5-11|   OOOOOOO. | 5-21|   OOOOOOO.
 | 5- 2|   OOOOOOOO | 5-12|   OOOOOO.. | 5-22|   OOOOOOOO
 | 5- 3|   OOOOOOO. | 5-13|   OOOOOOO. | 5-23|   OOOOOOOO
 | 5- 4|   OOOOOOO. | 5-14|   OOOOO... | 5-24|   OOOOOOOO
 | 5- 5|   OOOOOOO. | 5-15|   OOOOOOOO | 5-25|   OOOOOOO.
 | 5- 6|   OOOOOOO. | 5-16|   OOOOOOO. | 5-26|   OOOOOOO.
 | 5- 7|   OOOOOO.. | 5-17|   OOOOO... | 5-27|   OOOOOOOO
 | 5- 8|   OOOOOOO. | 5-18|   OOOOOOO. | 5-28|   OOOOOOOO
 | 5- 9|   OOOOOO.. | 5-19|   OOOOO... | 5-29|   OOOOOO..
 | 5-10|   OOOOOOO. | 5-20|   OOOOOOOO | 5-30|   OOOOOOOO


 | 9- 1|   O.......
 | 9- 2|   ........


 |10- 1|   OOO.....
 |10- 2|   OO......


Totals:
Total cores: 2912 
Total memory: 29.50 TiB 

Down:
Down cores: 0 (  0.00%) 
Down memory: 0.00 TiB (  0.00%) 

Used:
Used cores: 2193 ( 75.31%) 
Used memory: 18.26 TiB ( 61.91%) 

Utilization excluding down
CPU utilization:  75.31% (global)  72.97% (average per node)
Memory utilization:  61.91% (global)  61.32% (average per node)

Average per node utilization (excluding down):  85.54% 

[2019-12-05 15:42:04]

[2019-12-05 15:40:03]
[store03 s94n01] Controller 0
[store03 s94n01] VD: 0
[store03 s94n01] Virtual Drive  0 - RAID1 -  - Optimal
[store03 s94n01] 
[store03 s94n02] Controller 0
[store03 s94n02] VD: 0
[store03 s94n02] Virtual Drive  0 - RAID1 -  - Optimal
[store03 s94n02] 
[store03 s95n01] Controller 0
[store03 s95n01] VD: 0
[store03 s95n01] Virtual Drive  0 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n01] Virtual Drive  1 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n01] Virtual Drive  2 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n01] Virtual Drive  3 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n01] Virtual Drive  4 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n01] Virtual Drive  5 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n01] Virtual Drive  6 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n01] Controller 1
[store03 s95n01] VD: 0
[store03 s95n01] Virtual Drive  0 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n01] Virtual Drive  1 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n01] Virtual Drive  2 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n01] Virtual Drive  3 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n01] Virtual Drive  4 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n01] Virtual Drive  5 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n01] Virtual Drive  6 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n01] 
[store03 s95n02] Controller 0
[store03 s95n02] VD: 0
[store03 s95n02] Virtual Drive  0 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n02] Virtual Drive  1 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n02] Virtual Drive  2 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n02] Virtual Drive  3 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n02] Virtual Drive  4 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n02] Virtual Drive  5 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n02] Virtual Drive  6 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n02] Virtual Drive  7 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n02] Virtual Drive  8 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n02] Virtual Drive  9 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n02] Virtual Drive 10 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n02] Virtual Drive 11 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n02] Virtual Drive 12 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n02] Virtual Drive 13 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n02] 
[store03 s96n01] Controller 0
[store03 s96n01] VD: 7
[store03 s96n01] Virtual Drive  7 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n01] Virtual Drive  8 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n01] Virtual Drive  9 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n01] Virtual Drive 10 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n01] Virtual Drive 11 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n01] Virtual Drive 12 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n01] Virtual Drive 13 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n01] Controller 1
[store03 s96n01] VD: 0
[store03 s96n01] Virtual Drive  0 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n01] Virtual Drive  1 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n01] Virtual Drive  2 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n01] Virtual Drive  3 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n01] Virtual Drive  4 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n01] Virtual Drive  5 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n01] Virtual Drive  6 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n01] 
[store03 s96n02] Controller 0
[store03 s96n02] VD: 0
[store03 s96n02] Virtual Drive  0 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n02] Virtual Drive  1 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n02] Virtual Drive  2 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n02] Virtual Drive  3 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n02] Virtual Drive  4 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n02] Virtual Drive  5 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n02] Virtual Drive  6 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n02] Virtual Drive  7 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n02] Virtual Drive  8 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n02] Virtual Drive  9 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n02] Virtual Drive 10 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n02] Virtual Drive 11 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n02] Virtual Drive 12 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n02] Virtual Drive 13 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n02] 
[store03 s97n01] Controller 0
[store03 s97n01] VD: 0
[store03 s97n01] Virtual Drive  0 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n01] Virtual Drive  1 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n01] Virtual Drive  2 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n01] Virtual Drive  3 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n01] Virtual Drive  4 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n01] Virtual Drive  5 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n01] Virtual Drive  6 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n01] Controller 1
[store03 s97n01] VD: 0
[store03 s97n01] Virtual Drive  0 - RAID1 -  - Optimal
[store03 s97n01] Controller 2
[store03 s97n01] VD: 0
[store03 s97n01] Virtual Drive  0 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n01] Virtual Drive  1 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n01] Virtual Drive  2 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n01] Virtual Drive  3 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n01] Virtual Drive  4 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n01] Virtual Drive  5 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n01] Virtual Drive  6 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n01] 
[store03 s97n02] Controller 0
[store03 s97n02] VD: 0
[store03 s97n02] Virtual Drive  0 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n02] Virtual Drive  1 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n02] Virtual Drive  2 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n02] Virtual Drive  3 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n02] Virtual Drive  4 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n02] Virtual Drive  5 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n02] Virtual Drive  6 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n02] Controller 1
[store03 s97n02] VD: 0
[store03 s97n02] Virtual Drive  0 - RAID1 -  - Optimal
[store03 s97n02] Controller 2
[store03 s97n02] VD: 0
[store03 s97n02] Virtual Drive  0 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n02] Virtual Drive  1 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n02] Virtual Drive  2 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n02] Virtual Drive  3 - RAID6 - UU_UUUUUUUUU - Partially Degraded- Rebuild: 93% - ETA: 1 Hours 55 Minutes
[store03 s97n02] Virtual Drive  4 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n02] Virtual Drive  5 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n02] Virtual Drive  6 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n02] 
[store03 s98n01] Controller 0
[store03 s98n01] VD: 0
[store03 s98n01] Virtual Drive  0 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n01] Virtual Drive  1 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n01] Virtual Drive  2 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n01] Virtual Drive  3 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n01] Virtual Drive  4 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n01] Virtual Drive  5 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n01] Virtual Drive  6 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n01] Controller 1
[store03 s98n01] VD: 0
[store03 s98n01] Virtual Drive  0 - RAID1 -  - Optimal
[store03 s98n01] Controller 2
[store03 s98n01] VD: 0
[store03 s98n01] Virtual Drive  0 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n01] Virtual Drive  1 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n01] Virtual Drive  2 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n01] Virtual Drive  3 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n01] Virtual Drive  4 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n01] Virtual Drive  5 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n01] Virtual Drive  6 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n01] 
[store03 s98n02] Controller 0
[store03 s98n02] VD: 0
[store03 s98n02] Virtual Drive  0 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n02] Virtual Drive  1 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n02] Virtual Drive  2 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n02] Virtual Drive  3 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n02] Virtual Drive  4 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n02] Virtual Drive  5 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n02] Virtual Drive  6 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n02] Controller 1
[store03 s98n02] VD: 0
[store03 s98n02] Virtual Drive  0 - RAID1 -  - Optimal
[store03 s98n02] Controller 2
[store03 s98n02] VD: 0
[store03 s98n02] Virtual Drive  0 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n02] Virtual Drive  1 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n02] Virtual Drive  2 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n02] Virtual Drive  3 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n02] Virtual Drive  4 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n02] Virtual Drive  5 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n02] Virtual Drive  6 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n02] 
[2019-12-05 15:30:04]
    NodeID   TargetID         Pool           Total            Used            Free   Use% Usage Bar                    ITotal        IFree     %
    ======   ========         ====           =====            ====            ====   ==== =========                    ======        =====     =
      9501       3001       normal       80008.1GB       14931.4GB       65076.7GB    19% [####                ]      1490.3M      1488.7M  100%
      9501       3002       normal       80008.1GB       14595.0GB       65413.1GB    18% [####                ]      1490.3M      1488.7M  100%
      9501       3003       normal       80008.1GB       15777.2GB       64230.9GB    20% [####                ]      1490.3M      1488.7M  100%
      9501       3004       normal       80008.1GB       15805.1GB       64203.0GB    20% [####                ]      1490.3M      1488.7M  100%
      9501       3005       normal       80008.1GB       16848.4GB       63159.7GB    21% [####                ]      1490.3M      1488.7M  100%
      9501       3006       normal       80008.1GB       14769.7GB       65238.3GB    18% [####                ]      1490.3M      1488.7M  100%
      9501       3007    emergency       80008.1GB       11011.3GB       68996.8GB    14% [###                 ]      1490.3M      1489.0M  100%
      9501       3008       normal       80008.1GB       15642.4GB       64365.7GB    20% [####                ]      1490.3M      1488.7M  100%
      9501       3009       normal       80008.1GB       16364.1GB       63644.0GB    20% [####                ]      1490.3M      1488.7M  100%
      9501       3010       normal       80008.1GB       16495.9GB       63512.2GB    21% [####                ]      1490.3M      1488.7M  100%
      9501       3011       normal       80008.1GB       15496.9GB       64511.2GB    19% [####                ]      1490.3M      1488.7M  100%
      9501       3012       normal       80008.1GB       17300.4GB       62707.7GB    22% [####                ]      1490.3M      1488.7M  100%
      9501       3013       normal       80008.1GB       15792.4GB       64215.7GB    20% [####                ]      1490.3M      1488.7M  100%
      9502       3014       normal       80008.1GB       17374.2GB       62633.8GB    22% [####                ]      1490.3M      1488.7M  100%
      9502       3015       normal       80008.1GB       15218.9GB       64789.2GB    19% [####                ]      1490.3M      1488.7M  100%
      9502       3016       normal       80008.1GB       15232.2GB       64775.9GB    19% [####                ]      1490.3M      1488.8M  100%
      9502       3017       normal       80008.1GB       15855.4GB       64152.7GB    20% [####                ]      1490.3M      1488.7M  100%
      9502       3018       normal       80008.1GB       16249.4GB       63758.7GB    20% [####                ]      1490.3M      1488.7M  100%
      9502       3019       normal       80008.1GB       15974.9GB       64033.2GB    20% [####                ]      1490.3M      1488.7M  100%
      9502       3020       normal       80008.1GB       15082.4GB       64925.7GB    19% [####                ]      1490.3M      1488.7M  100%
      9601       3021       normal       80008.1GB       15108.4GB       64899.7GB    19% [####                ]      1490.3M      1488.7M  100%
      9601       3022       normal       80008.1GB       15338.4GB       64669.7GB    19% [####                ]      1490.3M      1488.7M  100%
      9601       3023       normal       80008.1GB       16103.0GB       63905.1GB    20% [####                ]      1490.3M      1488.7M  100%
      9601       3024       normal       80008.1GB       15337.4GB       64670.6GB    19% [####                ]      1490.3M      1488.7M  100%
      9601       3025       normal       80008.1GB       15710.8GB       64297.2GB    20% [####                ]      1490.3M      1488.7M  100%
      9601       3026       normal       80008.1GB       16078.6GB       63929.5GB    20% [####                ]      1490.3M      1488.7M  100%
      9601       3027       normal       80008.1GB       14944.1GB       65064.0GB    19% [####                ]      1490.3M      1488.7M  100%
      9601       3028       normal       80008.1GB       17212.2GB       62795.9GB    22% [####                ]      1490.3M      1488.7M  100%
      9601       3029       normal       80008.1GB       16638.3GB       63369.8GB    21% [####                ]      1490.3M      1488.7M  100%
      9601       3030       normal       80008.1GB       15919.3GB       64088.8GB    20% [####                ]      1490.3M      1488.7M  100%
      9601       3031       normal       80008.1GB       15441.6GB       64566.5GB    19% [####                ]      1490.3M      1488.7M  100%
      9601       3032       normal       80008.1GB       14880.3GB       65127.8GB    19% [####                ]      1490.3M      1488.7M  100%
      9601       3033       normal       80008.1GB       14681.3GB       65326.8GB    18% [####                ]      1490.3M      1488.7M  100%
      9601       3034       normal       80008.1GB       16078.0GB       63930.1GB    20% [####                ]      1490.3M      1488.7M  100%
      9602       3035       normal       80008.1GB       15178.7GB       64829.3GB    19% [####                ]      1490.3M      1488.7M  100%
      9602       3036       normal       80008.1GB       16451.7GB       63556.3GB    21% [####                ]      1490.3M      1488.7M  100%
      9602       3037       normal       80008.1GB       15883.5GB       64124.6GB    20% [####                ]      1490.3M      1488.7M  100%
      9602       3038       normal       80008.1GB       13791.4GB       66216.7GB    17% [###                 ]      1490.3M      1488.8M  100%
      9602       3039       normal       80008.1GB       16254.8GB       63753.3GB    20% [####                ]      1490.3M      1488.7M  100%
      9602       3040       normal       80008.1GB       15901.9GB       64106.2GB    20% [####                ]      1490.3M      1488.7M  100%
      9602       3041       normal       80008.1GB       15082.7GB       64925.4GB    19% [####                ]      1490.3M      1488.7M  100%
      9701       3042          low      119987.9GB       90100.4GB       29887.5GB    75% [###############     ]      2235.0M      2225.9M  100%
      9701       3043          low      119987.9GB       87948.7GB       32039.2GB    73% [###############     ]      2235.0M      2225.9M  100%
      9701       3044          low      119987.9GB       92453.8GB       27534.1GB    77% [###############     ]      2235.0M      2225.9M  100%
      9701       3045          low      119987.9GB       89837.2GB       30150.6GB    75% [###############     ]      2235.0M      2225.9M  100%
      9701       3046          low      119987.9GB       91046.3GB       28941.6GB    76% [###############     ]      2235.0M      2225.9M  100%
      9701       3047          low      119987.9GB       88455.2GB       31532.6GB    74% [###############     ]      2235.0M      2225.9M  100%
      9701       3048          low      119987.9GB       90511.0GB       29476.8GB    75% [###############     ]      2235.0M      2225.9M  100%
      9701       3049          low      119987.9GB       88961.6GB       31026.2GB    74% [###############     ]      2235.0M      2225.9M  100%
      9701       3050          low      119987.9GB       90504.7GB       29483.1GB    75% [###############     ]      2235.0M      2225.9M  100%
      9701       3051          low      119987.9GB       90574.6GB       29413.3GB    75% [###############     ]      2235.0M      2225.9M  100%
      9701       3052          low      119987.9GB       90910.3GB       29077.6GB    76% [###############     ]      2235.0M      2225.9M  100%
      9701       3053    emergency      119987.9GB       89301.5GB       30686.3GB    74% [###############     ]      2235.0M      2225.9M  100%
      9701       3054          low      119987.9GB       90233.2GB       29754.6GB    75% [###############     ]      2235.0M      2225.9M  100%
      9701       3055          low      119987.9GB       91169.3GB       28818.6GB    76% [###############     ]      2235.0M      2225.9M  100%
      9702       3056          low      119987.9GB       90243.7GB       29744.1GB    75% [###############     ]      2235.0M      2225.9M  100%
      9702       3057          low      119987.9GB       88840.4GB       31147.4GB    74% [###############     ]      2235.0M      2225.9M  100%
      9702       3058          low      119987.9GB       90705.0GB       29282.9GB    76% [###############     ]      2235.0M      2225.9M  100%
      9702       3059          low      119987.9GB       90141.3GB       29846.6GB    75% [###############     ]      2235.0M      2225.9M  100%
      9702       3060          low      119987.9GB       89318.9GB       30669.0GB    74% [###############     ]      2235.0M      2225.9M  100%
      9702       3061          low      119987.9GB       90770.6GB       29217.3GB    76% [###############     ]      2235.0M      2225.9M  100%
      9702       3062          low      119987.9GB       89180.2GB       30807.6GB    74% [###############     ]      2235.0M      2225.9M  100%
      9702       3063          low      119987.9GB       91226.7GB       28761.2GB    76% [###############     ]      2235.0M      2225.9M  100%
      9702       3064          low      119987.9GB       89080.4GB       30907.5GB    74% [###############     ]      2235.0M      2225.9M  100%
      9702       3065          low      119987.9GB       88385.4GB       31602.4GB    74% [###############     ]      2235.0M      2225.9M  100%
      9702       3066    emergency      119987.9GB       93030.3GB       26957.5GB    78% [################    ]      2235.0M      2225.9M  100%
      9702       3067          low      119987.9GB       90434.0GB       29553.9GB    75% [###############     ]      2235.0M      2225.9M  100%
      9702       3068          low      119987.9GB       89356.0GB       30631.9GB    74% [###############     ]      2235.0M      2225.9M  100%
      9702       3069          low      119987.9GB       88232.3GB       31755.6GB    74% [###############     ]      2235.0M      2225.9M  100%
      9801       3070          low      119987.9GB       88625.4GB       31362.5GB    74% [###############     ]      2235.0M      2225.9M  100%
      9801       3071          low      119987.9GB       72536.4GB       47451.4GB    60% [############        ]      2235.0M      2228.9M  100%
      9801       3072          low      119987.9GB       89835.8GB       30152.1GB    75% [###############     ]      2235.0M      2225.9M  100%
      9801       3073          low      119987.9GB       90153.4GB       29834.5GB    75% [###############     ]      2235.0M      2225.9M  100%
      9801       3074          low      119987.9GB       90938.4GB       29049.5GB    76% [###############     ]      2235.0M      2225.9M  100%
      9801       3075          low      119987.9GB       88874.3GB       31113.6GB    74% [###############     ]      2235.0M      2225.9M  100%
      9801       3076          low      119987.9GB       87724.5GB       32263.4GB    73% [###############     ]      2235.0M      2225.9M  100%
      9801       3077          low      119987.9GB       91128.7GB       28859.1GB    76% [###############     ]      2235.0M      2225.9M  100%
      9801       3078          low      119987.9GB       89754.7GB       30233.1GB    75% [###############     ]      2235.0M      2225.9M  100%
      9801       3079          low      119987.9GB       88752.5GB       31235.3GB    74% [###############     ]      2235.0M      2225.9M  100%
      9801       3080    emergency      119987.9GB       73512.0GB       46475.8GB    61% [############        ]      2235.0M      2228.8M  100%
      9801       3081          low      119987.9GB       90629.8GB       29358.1GB    76% [###############     ]      2235.0M      2225.9M  100%
      9801       3082          low      119987.9GB       89212.7GB       30775.1GB    74% [###############     ]      2235.0M      2225.9M  100%
      9801       3083          low      119987.9GB       91673.2GB       28314.7GB    76% [###############     ]      2235.0M      2225.9M  100%
      9802       3084          low      119987.9GB       89882.5GB       30105.4GB    75% [###############     ]      2235.0M      2225.9M  100%
      9802       3085          low      119987.9GB       91074.6GB       28913.2GB    76% [###############     ]      2235.0M      2225.9M  100%
      9802       3086          low      119987.9GB       90432.6GB       29555.3GB    75% [###############     ]      2235.0M      2225.9M  100%
      9802       3087          low      119987.9GB       88594.9GB       31392.9GB    74% [###############     ]      2235.0M      2225.9M  100%
      9802       3088          low      119987.9GB       90478.5GB       29509.4GB    75% [###############     ]      2235.0M      2225.9M  100%
      9802       3089          low      119987.9GB       91307.0GB       28680.9GB    76% [###############     ]      2235.0M      2225.9M  100%
      9802       3090          low      119987.9GB       88130.3GB       31857.6GB    73% [###############     ]      2235.0M      2225.9M  100%
      9802       3091          low      119987.9GB       88924.0GB       31063.9GB    74% [###############     ]      2235.0M      2225.9M  100%
      9802       3092          low      119987.9GB       89157.9GB       30830.0GB    74% [###############     ]      2235.0M      2225.9M  100%
      9802       3093          low      119987.9GB       88551.6GB       31436.3GB    74% [###############     ]      2235.0M      2225.9M  100%
      9802       3094          low      119987.9GB       90061.6GB       29926.3GB    75% [###############     ]      2235.0M      2225.9M  100%
      9802       3095          low      119987.9GB       89137.7GB       30850.2GB    74% [###############     ]      2235.0M      2225.9M  100%
      9802       3096          low      119987.9GB       90376.1GB       29611.8GB    75% [###############     ]      2235.0M      2225.9M  100%
      9802       3097          low      119987.9GB       89988.9GB       29999.0GB    75% [###############     ]      2235.0M      2225.9M  100%
      9502       3098       normal      119993.9GB       15758.1GB      104235.8GB    13% [###                 ]      2235.1M      2233.5M  100%
      9502       3099       normal      119993.9GB       16304.8GB      103689.1GB    14% [###                 ]      2235.1M      2233.5M  100%
      9502       3100       normal      119993.9GB       17273.6GB      102720.3GB    14% [###                 ]      2235.1M      2233.5M  100%
      9502       3101       normal      119993.9GB       16189.4GB      103804.5GB    13% [###                 ]      2235.1M      2233.5M  100%
      9502       3102       normal      119993.9GB       15575.8GB      104418.1GB    13% [###                 ]      2235.1M      2233.5M  100%
      9502       3103       normal      119993.9GB       15355.2GB      104638.7GB    13% [###                 ]      2235.1M      2233.5M  100%
      9502       3104       normal      119993.9GB       16982.0GB      103011.9GB    14% [###                 ]      2235.1M      2233.5M  100%
      9602       3105       normal      119993.9GB       16236.5GB      103757.4GB    14% [###                 ]      2235.1M      2233.5M  100%
      9602       3106       normal      119993.9GB       15854.2GB      104139.7GB    13% [###                 ]      2235.1M      2233.5M  100%
      9602       3107       normal      119993.9GB       15776.6GB      104217.3GB    13% [###                 ]      2235.1M      2233.5M  100%
      9602       3108       normal      119993.9GB       16399.3GB      103594.6GB    14% [###                 ]      2235.1M      2233.5M  100%
      9602       3109       normal      119993.9GB       16767.3GB      103226.6GB    14% [###                 ]      2235.1M      2233.5M  100%
      9602       3110       normal      119993.9GB       15772.0GB      104221.9GB    13% [###                 ]      2235.1M      2233.5M  100%
      9602       3111       normal      119993.9GB       15344.7GB      104649.1GB    13% [###                 ]      2235.1M      2233.5M  100%

Total size of /faststorage             11679565.5 GB
Total used space on /faststorage        5865825.9 GB
Total free space on /faststorage        5813739.6 GB

                                 Use%   50.22%

2019-12-05 15:43:06.073864377 +01:00
   SSSSSSSSSSSSSSS                                            iiii
 SS:::::::::::::::S                                          i::::i
S:::::SSSSSS::::::S                                           iiii
S:::::S     SSSSSSS
S:::::S            uuuuuu    uuuuuuvvvvvvv           vvvvvvviiiiiii
S:::::S            u::::u    u::::u v:::::v         v:::::v i:::::i
 S::::SSSS         u::::u    u::::u  v:::::v       v:::::v   i::::i
  SS::::::SSSSS    u::::u    u::::u   v:::::v     v:::::v    i::::i
    SSS::::::::SS  u::::u    u::::u    v:::::v   v:::::v     i::::i
       SSSSSS::::S u::::u    u::::u     v:::::v v:::::v      i::::i
            S:::::Su::::u    u::::u      v:::::v:::::v       i::::i
            S:::::Su:::::uuuu:::::u       v:::::::::v        i::::i
SSSSSSS     S:::::Su:::::::::::::::uu      v:::::::v        i::::::i
S::::::SSSSSS:::::S u:::::::::::::::u       v:::::v         i::::::i
S:::::::::::::::SS   uu::::::::uu:::u        v:::v          i::::::i
 SSSSSSSSSSSSSSS       uuuuuuuu  uuuu         vvv           iiiiiiii

                                        FreeBSD - The Power to Serve
[2019-12-03 00:34:04] s05n28 resumed
[2019-12-03 00:34:04] s05n18 resumed
[2019-12-03 00:34:04] s05n16 resumed
[2019-12-03 00:32:22] s05n28 now marked for drain: Prolog error 
[2019-12-03 00:32:22] s05n18 now marked for drain: Prolog error 
[2019-12-03 00:32:22] s05n16 now marked for drain: Prolog error 
[2019-12-03 00:30:22] s05n21 resumed
[2019-12-03 00:30:22] s05n20 resumed
[2019-12-03 00:28:08] s05n27 resumed
[2019-12-03 00:28:08] s05n21 now marked for drain: Prolog error 
[2019-12-03 00:28:08] s05n20 now marked for drain: Prolog error 
[2019-12-03 00:26:16] s05n27 now marked for drain: Prolog error 
[2019-12-03 00:26:16] s05n25 resumed
[2019-12-03 00:24:16] s05n25 now marked for drain: Prolog error 
[2019-12-03 00:20:21] s05n27 resumed
[2019-12-03 00:20:21] s05n19 resumed
[2019-12-03 00:20:21] s05n17 resumed
[2019-12-03 00:18:19] s05n27 now marked for drain: Prolog error 
[2019-12-03 00:18:19] s05n19 now marked for drain: Prolog error 
[2019-12-03 00:18:19] s05n17 now marked for drain: Prolog error 
[2019-12-03 00:16:02] s05n25 resumed
[2019-12-03 00:14:10] s05n25 now marked for drain: Prolog error 
[2019-12-03 00:14:10] s05n21 resumed
[2019-12-03 00:14:10] s05n19 resumed
[2019-12-03 00:12:22] s05n21 now marked for drain: Prolog error 
[2019-12-03 00:12:22] s05n19 now marked for drain: Prolog error 
[2019-12-03 00:10:28] s05n20 resumed
[2019-12-03 00:10:28] s05n19 resumed
[2019-12-03 00:08:13] s05n20 now marked for drain: Prolog error 
[2019-12-03 00:08:13] s05n19 now marked for drain: Prolog error 
[2019-12-02 21:34:02] s09n01 resumed
[2019-12-02 19:02:02] s09n01 now marked for drain: sanity-check: beegfs client Error storage targetID 
[2019-12-02 07:44:02] NOTICE !!! fs3 responding again via IB     
[2019-12-02 07:42:03] WARNING !!! fs3 not responding via IB     
[2019-11-30 13:04:02] NOTICE !!! fs3 responding again via IB     
[2019-11-30 13:02:03] WARNING !!! fs3 not responding via IB     
[2019-11-29 09:20:02] NOTICE !!! fs3 responding again via IB     
[2019-11-29 09:18:03] WARNING !!! fs3 not responding via IB     
[2019-11-28 22:28:11] s03n55 resumed
[2019-11-28 22:26:17] s03n55 now marked for drain: Prolog error 
[2019-11-28 20:08:03] s04n82 resumed
[2019-11-28 18:36:05] s04n82 now marked for drain: Not responding 
[2019-11-28 16:46:03] s03n36 resumed
[2019-11-28 16:44:12] s03n36 now marked for drain: Prolog error 
[2019-11-28 09:32:03] s03n48 resumed
[2019-11-28 08:22:03] s05n14 resumed
[2019-11-28 03:30:05] s05n14 now marked for drain: sanity-check: not enough space on /scratch - less than 100G 
[2019-11-28 03:06:02] s05n10 resumed
[2019-11-28 02:48:02] s05n10 now marked for drain: sanity-check: not enough space on /scratch - less than 100G 
[2019-11-28 00:32:06] s05n05 resumed
[2019-11-28 00:20:04] s05n02 resumed
[2019-12-05 15:00:03]
Backup on    fs1  finished 0-11:10:52 ago
Backup on    fs2  finished 0-04:14:27 ago
Backup on    fs3  finished 0-10:32:54 ago
Backup on    fs4  finished 2-20:30:37 ago
Backup on    fs5  finished 0-15:51:16 ago
Backup on    fs6  finished 0-10:54:40 ago
Backup on s95n01  finished 0-16:37:58 ago
[2019-12-05 15:42:04]
################
# RUNNING JOBS #
################
Count  USER      MIN_CPUS  MIN_MEMORY  TIMELIMIT
-----  ----      --------  ----------  ---------
1      andros    22        32G         11:55:00
1      au342309  16        128G        12:00:00
1      dario     1         50G         11:59:00
1      demontis  1         250000M     12:00:00
1      gualpo    1         64G         8:00:00
1      ijjduan   1         10G         4-00:00:00
1      iveera    1         4G          2-00:00:00
1      iveera    4         64G         1-00:00:00
1      lucie     16        64G         20:00:00
1      lucie     3         156G        10:00:00
1      mhc       1         20G         10:00:00
1      michaelk  1         1G          12:00:00
1      michaelk  1         8G          12:00:00
1      mrjensen  1         32G         1-06:00:00
1      mrjensen  1         32G         4:00:00
1      mtxellrb  1         100G        4:00:00
1      noor      1         100G        12:00:00
1      noor      12        160G        1-06:00:00
1      samuele   16        100G        2-00:00:00
1      sla       1         32G         1:00:00
1      yorgos    1         1G          4:00:00
2      anbt      16        64G         12:00:00
2      b960499   1         64G         2-00:00:00
2      iker      1         12G         2:00:00
2      inhs      1         100G        2-00:00:00
2      kalle     1         10G         15:00:00
2      marichim  10        32G         7-00:00:00
3      anksi     1         30G         1-16:00:00
3      ijjduan   1         100G        12:00:00
3      palle     8         16G         2-00:00:00
4      anksi     1         30G         2-00:00:00
6      carag     36        350G        12:00:00
19     agomez    16        32G         2:00:00
20     maretty   8         8G          12:00:00
29     jossti    16        1G          1-00:00:00
60     juraj     2         8G          16:00:00
77     juraj     8         16G         12:00:00
116    lucie     1         100G        1-16:00:00
###############
# QUEUED JOBS #
###############
Count  USER      MIN_CPUS  MIN_MEMORY  TIMELIMIT
-----  ----      --------  ----------  ---------
1      b960499   1         4G          4:00:00
1      demontis  24        300G        2-00:00:00
1      inagy     12        64G         12-12:00:00
1      lucie     10        20G         12:00:00
1      mrjensen  4         6G          5-00:00:00
1      zexi      1         10G         30-12:00:00
2      lucie     20        150G        10:00:00
2      marichim  10        32G         7-00:00:00
4      vang      8         128G        2-00:00:00
6      b960499   1         64G         9-00:00:00
10     b960499   1         64G         2-00:00:00
11     anbt      16        64G         12:00:00
14     lucie     24        256G        12:00:00
15     carag     36        350G        12:00:00
16     lucie     3         50G         1-00:00:00
19     lucie     3         156G        10:00:00
21     owyn      24        1T          10-00:00:00
60     maretty   8         16G         4-00:00:00
60     palle     8         16G         2-00:00:00
78     jossti    16        1G          1-00:00:00
110    iker      1         12G         2:00:00
110    lucie     1         100G        1-16:00:00
140    juraj     2         8G          16:00:00
196    ijjduan   1         100G        12:00:00
211    juraj     8         16G         12:00:00
300    iker      1         12G         10:00:00
1327   agomez    16        32G         2:00:00
 
#################
# CPUs per User #
#################
NCPUs  USER      QOS
-----  ----      ---
1      b960499   basic
1      b960499   bumped
1      dario     qos1
1      demontis  qos1
1      gualpo    basic
1      iveera    bumped
1      kalle     bumped
1      kalle     qos1
1      mhc       bumped
1      michaelk  bumped
1      michaelk  qos1
1      mtxellrb  bumped
1      sla       basic
1      yorgos    bumped
2      ijjduan   bumped
2      ijjduan   qos1
2      iker      bumped
2      inhs      qos1
2      mrjensen  bumped
4      iveera    qos1
7      anksi     bumped
13     noor      bumped
16     au342309  bumped
16     samuele   basic
20     marichim  basic
22     andros    qos1
24     palle     qos1
32     anbt      bumped
36     carag     bumped
135    lucie     qos1
160    maretty   qos1
180    carag     basic
304    agomez    qos1
464    jossti    basic
736    juraj     basic
########################
# QUEUED CPUs per User #
########################
NCPUs  USER      QOS
-----  ----      ---
1      zexi      basic
4      b960499   bumped
4      mrjensen  bumped
8      palle     bumped
12     inagy     qos1
13     b960499   basic
20     marichim  basic
24     demontis  bumped
29     iker      bumped
32     maretty   bumped
32     vang      gold
176    anbt      basic
196    ijjduan   qos1
381    iker      basic
448    maretty   qos1
472    palle     qos1
504    owyn      basic
540    carag     basic
601    lucie     qos1
1248   jossti    basic
1968   juraj     basic
21232  agomez    qos1
[2019-12-05 15:42:04]
### Node diagnostics (experimental) ###