[2020-04-08 05:36:02]
         fs1  fs3  fs5  fs6
1  Gig   UP   UP   UP   UP 
10 Gig   n/a  UP   UP   UP 
IB       n/a  UP   UP   UP 
[2020-04-08 05:36:03]

 | 1-11|   OOOOOOOO | 1-21|   OOOOOOOO | 1-31|   OOOOOOOO | 1-41|   OOOOOOOO | 1-51|   OOOOOOOO | 1-61|   OOOOOOOO
 | 1-12|   OOOOOOOO | 1-22|   OOOOOOOO | 1-32|   OOOOOOOO | 1-42|   OOOOOOOO | 1-52|   OOOOOOOO | 1-62|   OOOOOOOO
 | 1-13|   OOOOOOOO | 1-23|   OOOOOOOO | 1-33|   OOOOOOOO | 1-43|   OOOOOOOO | 1-53|   OOOOOOOO | 1-63|   OOOOOOOO
 | 1-14|   OOOOOOOO | 1-24|   OOOOOOOO | 1-34|   OOOOOOOO | 1-44|   OOOOOOOO | 1-54|   OOOOOOOO | 1-64|   OOOOOOOO
 | 1-15|   OOOOOOOO | 1-25|   OOOOOOOO | 1-35|   OOOOOOOO | 1-45|   OOOOOOOO | 1-55|   OOOOOOOO | 1-65|   OOOOOOOO
 | 1-16|   OOOOOOOO | 1-26|   OOOOOOOO | 1-36|   OOOOOOOO | 1-46|   OOOOOOOO | 1-56|   OOOOOOOO | 1-66|   OOOOOOOO
 | 1-17|   OOOOOOOO | 1-27|   OOOOOOOO | 1-37|   OOOOOOOO | 1-47|   OOOOOOOO | 1-57|   OOOOOOOO | 1-67|   OOOOOOOO
 | 1-18|   OOOOOOOO | 1-28|   OOOOOOOO | 1-38|   OOOOOOOO | 1-48|   OOOOOOOO | 1-58|   OOOOOOOO                   


 | 3-11|   OOOOOOOO | 3-21|   OOOOOOOO | 3-31|   OOOOOOOO | 3-41|   OOOOOOOO | 3-51|   OOOOOOO. | 3-61|   OOOOOOO. | 3-71|   OOOOOOO.
 | 3-12|   OOOOOOOO | 3-22|   OOOOOOOO | 3-32|   OOOOOOOO | 3-42|   OOOOOOOO | 3-52|     DOWN   | 3-62|   OOOOOO.. | 3-72|     DOWN  
 | 3-13|   OOOOOOOO | 3-23|   OOOOOOOO | 3-33|   OOOOOOOO | 3-43|   OOOOOOOO | 3-53|   OOOOOOO. | 3-63|   OOOOOOO. | 3-73|   OOOOOOO.
 | 3-14|   OOOOOOOO | 3-24|   OOOOOOOO | 3-34|   OOOOOOOO | 3-44|   OOOOOOOO | 3-54|   OOOO.... | 3-64|   OOOOOOOO | 3-74|   OOOOOOO.
 | 3-15|   OOOOOOO. | 3-25|   OOOOOOOO | 3-35|   OOOOOOOO | 3-45|   OOOOOOO. | 3-55|   OOOOOOO. | 3-65|   OOOOOOO. | 3-75|   OOOOOOO.
 | 3-16|   OOOOOOOO | 3-26|   OOOOOOOO | 3-36|   OOOOOOOO | 3-46|   OOOOOOO. | 3-56|   OOOOOOOO | 3-66|   OOOOOOO. | 3-76|   OOOOOOO.
 | 3-17|   OOOOOOOO | 3-27|   OOOOOOOO | 3-37|   OOOOOOOO | 3-47|   OOOOOO.. | 3-57|   OOOOOO.. | 3-67|   OOOOOOO. | 3-77|   OOOOOOO.
 | 3-18|   OOOOOOOO | 3-28|   OOOOOOOO | 3-38|   OOOOOOOO | 3-48|   OOOOOO.. | 3-58|   OOOOOO.. | 3-68|   OOOOOOO. | 3-78|   ........


 | 4-11|   OOOOOOOO | 4-21|   OOOOOO.. | 4-31|   OOOOOO.. | 4-41|   OOOOOOOO | 4-51|   OOOOOOOO | 4-61|   OOOOOOOO | 4-71|   OOOOOOOO | 4-81|   OOOOOOOO
 | 4-12|   OOOOOOO. | 4-22|   OOOOOO.. | 4-32|   OOOOOO.. | 4-42|   OOOOOOOO | 4-52|   OOOOOOO. | 4-62|   OOOOOOO. | 4-72|   OOOOOOO. | 4-82|   OOOOOOOO
 | 4-13|   OOOOOOO. | 4-23|   OOOOOO.. | 4-33|   OOOOOO.. | 4-43|   OOOOOOOO | 4-53|   OOOOOOOO | 4-63|   OOOOOOOO | 4-73|   OOOOOOOO | 4-83|   OOOOOOOO
 | 4-14|   OOOOOOOO | 4-24|   OOOOOO.. | 4-34|   OOOOOO.. | 4-44|   OOOOOOOO | 4-54|   OOOOOOOO | 4-64|   OOOOOOOO | 4-74|   OOOOOOOO                   


 | 5- 1|   OOOOOOO. | 5-11|   OOOOOOO. | 5-21|   OOOOOOO.
 | 5- 2|   OOOOOOO. | 5-12|   OOOOOOO. | 5-22|   OOOOOOO.
 | 5- 3|   OOOOOOO. | 5-13|   OOOOOOO. | 5-23|   OOOOOOO.
 | 5- 4|   OOOOOOO. | 5-14|   OOOOOOO. | 5-24|   OOOOOOO.
 | 5- 5|   OOOOOOO. | 5-15|   OOOOOOO. | 5-25|   OOOOOOO.
 | 5- 6|   OOOOOOO. | 5-16|   OOOOO... | 5-26|   OOOOOOO.
 | 5- 7|   OOOOOOO. | 5-17|   OOOOO... | 5-27|   OOOOOO..
 | 5- 8|   OOOOOOO. | 5-18|   OOOOOOOO | 5-28|   OOOOOOO.
 | 5- 9|   OOOOOOO. | 5-19|   OOOOO... | 5-29|   OOOOOOOO
 | 5-10|   OOOOOOO. | 5-20|   OOOOO... | 5-30|   OOOOO...


 | 9- 1|   OOOOOOOO
 | 9- 2|   OOOOOOOO
 | 9- 3|   ........


 |10- 1|   ........
 |10- 2|   ........


 |91- 1|   OOOOOOOO
 |91- 2|   OOOOOOOO
 |91- 3|   OOOOOOOO
 |91- 4|     DOWN  
 |91- 5|   OOOOOOOO
 |91- 6|   OOOOOOOO


Totals:
Total cores: 3736 
Total memory: 36.38 TiB 

Down:
Down cores: 52 (  1.39%) 
Down memory: 0.38 TiB (  1.03%) 

Used:
Used cores: 3111 ( 83.27%) 
Used memory: 20.23 TiB ( 55.61%) 

Utilization excluding down
CPU utilization:  84.45% (global)  85.07% (average per node)
Memory utilization:  56.18% (global)  48.90% (average per node)

Average per node utilization (excluding down):  94.10% 
[2020-04-08 05:36:03]
s03n52 Looks like a loose 1G cable... after that a reboot to make sure beegfs and network are fine. (had some problems before the loose cable) 
s03n72 beegfs restart 
s91n04 Node unexpectedly rebooted 

[2020-04-08 05:35:14]
[store03 s94n01] Controller 0
[store03 s94n01] Virtual Drive  0 - RAID1 - UU - Optimal
[store03 s94n01] 

[store03 s94n02] Controller 0
[store03 s94n02] Virtual Drive  0 - RAID1 - UU - Optimal
[store03 s94n02] 

[store03 s95n01] Controller 0
[store03 s95n01] Virtual Drive  0 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n01] Virtual Drive  1 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n01] Virtual Drive  2 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n01] Virtual Drive  3 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n01] Virtual Drive  4 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n01] Virtual Drive  5 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n01] Virtual Drive  6 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n01] Controller 1
[store03 s95n01] Virtual Drive  0 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n01] Virtual Drive  1 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n01] Virtual Drive  2 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n01] Virtual Drive  3 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n01] Virtual Drive  4 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n01] Virtual Drive  5 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n01] Virtual Drive  6 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n01] 

[store03 s95n02] Controller 0
[store03 s95n02] Virtual Drive  0 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n02] Virtual Drive  1 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n02] Virtual Drive  2 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n02] Virtual Drive  3 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n02] Virtual Drive  4 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n02] Virtual Drive  5 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n02] Virtual Drive  6 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n02] Virtual Drive  7 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n02] Virtual Drive  8 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n02] Virtual Drive  9 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n02] Virtual Drive 10 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n02] Virtual Drive 11 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n02] Virtual Drive 12 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n02] Virtual Drive 13 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s95n02] 

[store03 s96n01] Controller 0
[store03 s96n01] Virtual Drive  7 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n01] Virtual Drive  8 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n01] Virtual Drive  9 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n01] Virtual Drive 10 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n01] Virtual Drive 11 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n01] Virtual Drive 12 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n01] Virtual Drive 13 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n01] Controller 1
[store03 s96n01] Virtual Drive  0 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n01] Virtual Drive  1 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n01] Virtual Drive  2 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n01] Virtual Drive  3 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n01] Virtual Drive  4 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n01] Virtual Drive  5 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n01] Virtual Drive  6 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n01] 

[store03 s96n02] Controller 0
[store03 s96n02] Virtual Drive  0 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n02] Virtual Drive  1 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n02] Virtual Drive  2 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n02] Virtual Drive  3 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n02] Virtual Drive  4 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n02] Virtual Drive  5 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n02] Virtual Drive  6 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n02] Virtual Drive  7 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n02] Virtual Drive  8 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n02] Virtual Drive  9 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n02] Virtual Drive 10 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n02] Virtual Drive 11 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n02] Virtual Drive 12 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n02] Virtual Drive 13 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s96n02] 

[store03 s97n01] Controller 0
[store03 s97n01] Virtual Drive  0 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n01] Virtual Drive  1 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n01] Virtual Drive  2 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n01] Virtual Drive  3 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n01] Virtual Drive  4 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n01] Virtual Drive  5 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n01] Virtual Drive  6 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n01] Controller 1
[store03 s97n01] Virtual Drive  0 - RAID1 - UU - Optimal
[store03 s97n01] Controller 2
[store03 s97n01] Virtual Drive  0 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n01] Virtual Drive  1 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n01] Virtual Drive  2 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n01] Virtual Drive  3 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n01] Virtual Drive  4 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n01] Virtual Drive  5 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n01] Virtual Drive  6 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n01] 

[store03 s97n02] Controller 0
[store03 s97n02] Virtual Drive  0 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n02] Virtual Drive  1 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n02] Virtual Drive  2 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n02] Virtual Drive  3 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n02] Virtual Drive  4 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n02] Virtual Drive  5 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n02] Virtual Drive  6 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n02] Controller 1
[store03 s97n02] Virtual Drive  0 - RAID1 - UU - Optimal
[store03 s97n02] Controller 2
[store03 s97n02] Virtual Drive  0 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n02] Virtual Drive  1 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n02] Virtual Drive  2 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n02] Virtual Drive  3 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n02] Virtual Drive  4 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n02] Virtual Drive  5 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n02] Virtual Drive  6 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s97n02] 

[store03 s98n01] Controller 0
[store03 s98n01] Virtual Drive  0 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n01] Virtual Drive  1 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n01] Virtual Drive  2 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n01] Virtual Drive  3 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n01] Virtual Drive  4 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n01] Virtual Drive  5 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n01] Virtual Drive  6 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n01] Controller 1
[store03 s98n01] Virtual Drive  0 - RAID1 - UU - Optimal
[store03 s98n01] Controller 2
[store03 s98n01] Virtual Drive  0 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n01] Virtual Drive  1 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n01] Virtual Drive  2 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n01] Virtual Drive  3 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n01] Virtual Drive  4 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n01] Virtual Drive  5 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n01] Virtual Drive  6 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n01] 

[store03 s98n02] Controller 0
[store03 s98n02] Virtual Drive  0 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n02] Virtual Drive  1 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n02] Virtual Drive  2 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n02] Virtual Drive  3 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n02] Virtual Drive  4 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n02] Virtual Drive  5 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n02] Virtual Drive  6 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n02] Controller 1
[store03 s98n02] Virtual Drive  0 - RAID1 - UU - Optimal
[store03 s98n02] Controller 2
[store03 s98n02] Virtual Drive  0 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n02] Virtual Drive  1 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n02] Virtual Drive  2 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n02] Virtual Drive  3 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n02] Virtual Drive  4 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n02] Virtual Drive  5 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n02] Virtual Drive  6 - RAID6 - UUUUUUUUUUUU - Optimal
[store03 s98n02] 

[2020-04-08 05:30:04]
    NodeID   TargetID         Pool           Total            Used            Free   Use% Usage Bar                    ITotal        IFree     %
    ======   ========         ====           =====            ====            ====   ==== =========                    ======        =====     =
      9501       3001       normal       80008.1GB       66786.0GB       13222.1GB    83% [#################   ]      1490.3M      1488.2M  100%
      9501       3002       normal       80008.1GB       67039.9GB       12968.2GB    84% [#################   ]      1490.3M      1488.2M  100%
      9501       3003       normal       80008.1GB       66848.5GB       13159.5GB    84% [#################   ]      1490.3M      1488.2M  100%
      9501       3004       normal       80008.1GB       67309.3GB       12698.8GB    84% [#################   ]      1490.3M      1488.2M  100%
      9501       3005       normal       80008.1GB       66814.8GB       13193.3GB    84% [#################   ]      1490.3M      1488.2M  100%
      9501       3006       normal       80008.1GB       67258.2GB       12749.9GB    84% [#################   ]      1490.3M      1488.2M  100%
      9501       3007       normal       80008.1GB       66609.2GB       13398.9GB    83% [#################   ]      1490.3M      1488.3M  100%
      9501       3008       normal       80008.1GB       66861.7GB       13146.3GB    84% [#################   ]      1490.3M      1488.2M  100%
      9501       3009       normal       80008.1GB       67019.2GB       12988.9GB    84% [#################   ]      1490.3M      1488.2M  100%
      9501       3010       normal       80008.1GB       67244.1GB       12763.9GB    84% [#################   ]      1490.3M      1488.2M  100%
      9501       3011       normal       80008.1GB       66958.3GB       13049.7GB    84% [#################   ]      1490.3M      1488.1M  100%
      9501       3012       normal       80008.1GB       66810.9GB       13197.2GB    84% [#################   ]      1490.3M      1488.1M  100%
      9501       3013       normal       80008.1GB       67258.8GB       12749.3GB    84% [#################   ]      1490.3M      1488.1M  100%
      9502       3014       normal       80008.1GB       66841.7GB       13166.3GB    84% [#################   ]      1490.3M      1488.1M  100%
      9502       3015       normal       80008.1GB       66936.7GB       13071.4GB    84% [#################   ]      1490.3M      1488.1M  100%
      9502       3016       normal       80008.1GB       67003.3GB       13004.8GB    84% [#################   ]      1490.3M      1490.1M  100%
      9502       3017       normal       80008.1GB       66761.5GB       13246.6GB    83% [#################   ]      1490.3M      1488.1M  100%
      9502       3018       normal       80008.1GB       66981.7GB       13026.3GB    84% [#################   ]      1490.3M      1488.1M  100%
      9502       3019       normal       80008.1GB       66980.9GB       13027.2GB    84% [#################   ]      1490.3M      1488.1M  100%
      9502       3020       normal       80008.1GB       66926.3GB       13081.8GB    84% [#################   ]      1490.3M      1488.1M  100%
      9601       3021       normal       80008.1GB       66698.0GB       13310.1GB    83% [#################   ]      1490.3M      1488.1M  100%
      9601       3022       normal       80008.1GB       66433.9GB       13574.2GB    83% [#################   ]      1490.3M      1488.1M  100%
      9601       3023       normal       80008.1GB       66937.7GB       13070.4GB    84% [#################   ]      1490.3M      1488.2M  100%
      9601       3024       normal       80008.1GB       67172.6GB       12835.4GB    84% [#################   ]      1490.3M      1488.2M  100%
      9601       3025       normal       80008.1GB       67140.2GB       12867.9GB    84% [#################   ]      1490.3M      1488.2M  100%
      9601       3026       normal       80008.1GB       66692.1GB       13316.0GB    83% [#################   ]      1490.3M      1488.2M  100%
      9601       3027       normal       80008.1GB       67060.1GB       12947.9GB    84% [#################   ]      1490.3M      1488.2M  100%
      9601       3028       normal       80008.1GB       66934.2GB       13073.8GB    84% [#################   ]      1490.3M      1488.2M  100%
      9601       3029       normal       80008.1GB       66719.0GB       13289.1GB    83% [#################   ]      1490.3M      1488.2M  100%
      9601       3030       normal       80008.1GB       67065.0GB       12943.1GB    84% [#################   ]      1490.3M      1488.2M  100%
      9601       3031       normal       80008.1GB       66931.3GB       13076.8GB    84% [#################   ]      1490.3M      1488.2M  100%
      9601       3032       normal       80008.1GB       66529.6GB       13478.5GB    83% [#################   ]      1490.3M      1488.2M  100%
      9601       3033       normal       80008.1GB       66808.8GB       13199.3GB    84% [#################   ]      1490.3M      1488.2M  100%
      9601       3034       normal       80008.1GB       67119.7GB       12888.3GB    84% [#################   ]      1490.3M      1488.2M  100%
      9602       3035       normal       80008.1GB       66766.9GB       13241.2GB    83% [#################   ]      1490.3M      1488.2M  100%
      9602       3036       normal       80008.1GB       67130.0GB       12878.0GB    84% [#################   ]      1490.3M      1488.2M  100%
      9602       3037       normal       80008.1GB       66905.7GB       13102.4GB    84% [#################   ]      1490.3M      1488.2M  100%
      9602       3038       normal       80008.1GB       66847.1GB       13160.9GB    84% [#################   ]      1490.3M      1488.3M  100%
      9602       3039       normal       80008.1GB       67200.8GB       12807.3GB    84% [#################   ]      1490.3M      1488.2M  100%
      9602       3040       normal       80008.1GB       67097.4GB       12910.7GB    84% [#################   ]      1490.3M      1488.3M  100%
      9602       3041       normal       80008.1GB       66904.2GB       13103.9GB    84% [#################   ]      1490.3M      1488.2M  100%
      9701       3042       normal      119987.9GB       66866.5GB       53121.4GB    56% [###########         ]      2235.0M      2226.2M  100%
      9701       3043       normal      119987.9GB       67087.8GB       52900.0GB    56% [###########         ]      2235.0M      2226.2M  100%
      9701       3044       normal      119987.9GB       67113.3GB       52874.5GB    56% [###########         ]      2235.0M      2226.2M  100%
      9701       3045       normal      119987.9GB       67074.3GB       52913.5GB    56% [###########         ]      2235.0M      2226.2M  100%
      9701       3046       normal      119987.9GB       66941.8GB       53046.1GB    56% [###########         ]      2235.0M      2226.2M  100%
      9701       3047       normal      119987.9GB       66933.7GB       53054.1GB    56% [###########         ]      2235.0M      2226.2M  100%
      9701       3048       normal      119987.9GB       67092.4GB       52895.5GB    56% [###########         ]      2235.0M      2226.2M  100%
      9701       3049       normal      119987.9GB       66922.3GB       53065.5GB    56% [###########         ]      2235.0M      2226.2M  100%
      9701       3050       normal      119987.9GB       67211.8GB       52776.0GB    56% [###########         ]      2235.0M      2226.2M  100%
      9701       3051       normal      119987.9GB       66894.1GB       53093.8GB    56% [###########         ]      2235.0M      2226.2M  100%
      9701       3052       normal      119987.9GB       66900.2GB       53087.7GB    56% [###########         ]      2235.0M      2226.2M  100%
      9701       3053       normal      119987.9GB       66887.6GB       53100.2GB    56% [###########         ]      2235.0M      2226.2M  100%
      9701       3054       normal      119987.9GB       66847.9GB       53140.0GB    56% [###########         ]      2235.0M      2226.2M  100%
      9701       3055       normal      119987.9GB       66954.7GB       53033.2GB    56% [###########         ]      2235.0M      2226.2M  100%
      9702       3056       normal      119987.9GB       66930.6GB       53057.2GB    56% [###########         ]      2235.0M      2226.2M  100%
      9702       3057       normal      119987.9GB       66889.8GB       53098.0GB    56% [###########         ]      2235.0M      2226.2M  100%
      9702       3058       normal      119987.9GB       66998.1GB       52989.8GB    56% [###########         ]      2235.0M      2226.2M  100%
      9702       3059       normal      119987.9GB       67318.6GB       52669.3GB    56% [###########         ]      2235.0M      2226.2M  100%
      9702       3060       normal      119987.9GB       67036.5GB       52951.3GB    56% [###########         ]      2235.0M      2226.2M  100%
      9702       3061       normal      119987.9GB       67000.9GB       52986.9GB    56% [###########         ]      2235.0M      2226.2M  100%
      9702       3062       normal      119987.9GB       66801.6GB       53186.2GB    56% [###########         ]      2235.0M      2226.2M  100%
      9702       3063       normal      119987.9GB       66896.7GB       53091.2GB    56% [###########         ]      2235.0M      2226.2M  100%
      9702       3064       normal      119987.9GB       66809.6GB       53178.2GB    56% [###########         ]      2235.0M      2226.2M  100%
      9702       3065       normal      119987.9GB       66785.1GB       53202.8GB    56% [###########         ]      2235.0M      2226.2M  100%
      9702       3066       normal      119987.9GB       67201.4GB       52786.5GB    56% [###########         ]      2235.0M      2226.2M  100%
      9702       3067       normal      119987.9GB       66784.3GB       53203.5GB    56% [###########         ]      2235.0M      2226.2M  100%
      9702       3068       normal      119987.9GB       66909.8GB       53078.1GB    56% [###########         ]      2235.0M      2226.2M  100%
      9702       3069       normal      119987.9GB       67078.0GB       52909.8GB    56% [###########         ]      2235.0M      2226.2M  100%
      9801       3070       normal      119987.9GB       66921.7GB       53066.1GB    56% [###########         ]      2235.0M      2226.2M  100%
      9801       3071       normal      119987.9GB       67068.3GB       52919.6GB    56% [###########         ]      2235.0M      2229.1M  100%
      9801       3072       normal      119987.9GB       67108.4GB       52879.5GB    56% [###########         ]      2235.0M      2226.2M  100%
      9801       3073       normal      119987.9GB       66940.3GB       53047.6GB    56% [###########         ]      2235.0M      2226.2M  100%
      9801       3074       normal      119987.9GB       66863.0GB       53124.9GB    56% [###########         ]      2235.0M      2226.2M  100%
      9801       3075       normal      119987.9GB       66980.6GB       53007.2GB    56% [###########         ]      2235.0M      2226.2M  100%
      9801       3076       normal      119987.9GB       67286.8GB       52701.0GB    56% [###########         ]      2235.0M      2226.2M  100%
      9801       3077       normal      119987.9GB       66778.8GB       53209.1GB    56% [###########         ]      2235.0M      2226.2M  100%
      9801       3078       normal      119987.9GB       66899.6GB       53088.3GB    56% [###########         ]      2235.0M      2226.2M  100%
      9801       3079       normal      119987.9GB       66899.5GB       53088.4GB    56% [###########         ]      2235.0M      2226.2M  100%
      9801       3080       normal      119987.9GB       66852.8GB       53135.0GB    56% [###########         ]      2235.0M      2229.0M  100%
      9801       3081       normal      119987.9GB       66986.4GB       53001.5GB    56% [###########         ]      2235.0M      2226.2M  100%
      9801       3082       normal      119987.9GB       66900.8GB       53087.1GB    56% [###########         ]      2235.0M      2226.2M  100%
      9801       3083       normal      119987.9GB       66858.0GB       53129.8GB    56% [###########         ]      2235.0M      2226.2M  100%
      9802       3084       normal      119987.9GB       66990.2GB       52997.6GB    56% [###########         ]      2235.0M      2226.2M  100%
      9802       3085       normal      119987.9GB       67011.0GB       52976.9GB    56% [###########         ]      2235.0M      2226.2M  100%
      9802       3086       normal      119987.9GB       67045.1GB       52942.8GB    56% [###########         ]      2235.0M      2226.2M  100%
      9802       3087       normal      119987.9GB       67068.3GB       52919.6GB    56% [###########         ]      2235.0M      2226.2M  100%
      9802       3088       normal      119987.9GB       67055.8GB       52932.0GB    56% [###########         ]      2235.0M      2226.2M  100%
      9802       3089       normal      119987.9GB       67085.4GB       52902.5GB    56% [###########         ]      2235.0M      2226.2M  100%
      9802       3090       normal      119987.9GB       66928.3GB       53059.6GB    56% [###########         ]      2235.0M      2226.2M  100%
      9802       3091       normal      119987.9GB       66696.8GB       53291.1GB    56% [###########         ]      2235.0M      2226.2M  100%
      9802       3092       normal      119987.9GB       66864.2GB       53123.7GB    56% [###########         ]      2235.0M      2226.2M  100%
      9802       3093       normal      119987.9GB       66832.6GB       53155.3GB    56% [###########         ]      2235.0M      2226.2M  100%
      9802       3094       normal      119987.9GB       66855.7GB       53132.2GB    56% [###########         ]      2235.0M      2226.2M  100%
      9802       3095       normal      119987.9GB       66799.3GB       53188.5GB    56% [###########         ]      2235.0M      2226.2M  100%
      9802       3096       normal      119987.9GB       67094.4GB       52893.5GB    56% [###########         ]      2235.0M      2226.2M  100%
      9802       3097       normal      119987.9GB       67008.7GB       52979.2GB    56% [###########         ]      2235.0M      2226.2M  100%
      9502       3098       normal      119993.9GB       66935.7GB       53058.2GB    56% [###########         ]      2235.1M      2233.0M  100%
      9502       3099       normal      119993.9GB       66914.1GB       53079.8GB    56% [###########         ]      2235.1M      2233.0M  100%
      9502       3100       normal      119993.9GB       67078.0GB       52915.9GB    56% [###########         ]      2235.1M      2233.0M  100%
      9502       3101       normal      119993.9GB       67006.2GB       52987.7GB    56% [###########         ]      2235.1M      2233.0M  100%
      9502       3102       normal      119993.9GB       66809.6GB       53184.3GB    56% [###########         ]      2235.1M      2233.0M  100%
      9502       3103       normal      119993.9GB       67486.5GB       52507.4GB    56% [###########         ]      2235.1M      2233.0M  100%
      9502       3104       normal      119993.9GB       66803.1GB       53190.8GB    56% [###########         ]      2235.1M      2233.0M  100%
      9602       3105       normal      119993.9GB       66741.4GB       53252.5GB    56% [###########         ]      2235.1M      2233.0M  100%
      9602       3106       normal      119993.9GB       67112.7GB       52881.2GB    56% [###########         ]      2235.1M      2233.0M  100%
      9602       3107       normal      119993.9GB       66781.9GB       53212.0GB    56% [###########         ]      2235.1M      2233.0M  100%
      9602       3108       normal      119993.9GB       66741.8GB       53252.1GB    56% [###########         ]      2235.1M      2233.0M  100%
      9602       3109       normal      119993.9GB       66804.8GB       53189.1GB    56% [###########         ]      2235.1M      2233.0M  100%
      9602       3110       normal      119993.9GB       66881.8GB       53112.1GB    56% [###########         ]      2235.1M      2233.0M  100%
      9602       3111       normal      119993.9GB       66903.9GB       53090.0GB    56% [###########         ]      2235.1M      2233.0M  100%

Total size of /faststorage             11679565.5 GB
Total used space on /faststorage        7431197.0 GB
Total free space on /faststorage        4248368.5 GB

                                 Use%   63.63%

2020-04-08 05:37:06.462326507 +02:00
   SSSSSSSSSSSSSSS                                            iiii
 SS:::::::::::::::S                                          i::::i
S:::::SSSSSS::::::S                                           iiii
S:::::S     SSSSSSS
S:::::S            uuuuuu    uuuuuuvvvvvvv           vvvvvvviiiiiii
S:::::S            u::::u    u::::u v:::::v         v:::::v i:::::i
 S::::SSSS         u::::u    u::::u  v:::::v       v:::::v   i::::i
  SS::::::SSSSS    u::::u    u::::u   v:::::v     v:::::v    i::::i
    SSS::::::::SS  u::::u    u::::u    v:::::v   v:::::v     i::::i
       SSSSSS::::S u::::u    u::::u     v:::::v v:::::v      i::::i
            S:::::Su::::u    u::::u      v:::::v:::::v       i::::i
            S:::::Su:::::uuuu:::::u       v:::::::::v        i::::i
SSSSSSS     S:::::Su:::::::::::::::uu      v:::::::v        i::::::i
S::::::SSSSSS:::::S u:::::::::::::::u       v:::::v         i::::::i
S:::::::::::::::SS   uu::::::::uu:::u        v:::v          i::::::i
 SSSSSSSSSSSSSSS       uuuuuuuu  uuuu         vvv           iiiiiiii

                                        FreeBSD - The Power to Serve
[2020-04-07 13:42:03] s91n04 now marked for drain: Node unexpectedly rebooted 
[2020-04-07 08:16:03] s91n04 resumed
[2020-04-07 08:12:03] s91n04 now marked for drain: Not responding 
[2020-04-07 01:32:02] s05n06 resumed
[2020-04-07 00:26:02] s05n07 resumed
[2020-04-06 23:06:02] s05n05 resumed
[2020-04-06 23:04:02] s01n24 resumed
[2020-04-06 23:02:02] s05n13 resumed
[2020-04-06 23:02:02] s01n24 now marked for drain: sanity-check: can't ping gfs3 
[2020-04-06 20:46:02] s05n06 now marked for drain: sanity-check: not enough space on /scratch - less than 100G 
[2020-04-06 20:44:03] s05n06 resumed
[2020-04-06 20:34:02] s05n07 now marked for drain: sanity-check: not enough space on /scratch - less than 100G 
[2020-04-06 20:32:02] s05n07 resumed
[2020-04-06 20:28:02] s05n13 now marked for drain: sanity-check: not enough space on /scratch - less than 100G 
[2020-04-06 20:26:02] s05n13 resumed
[2020-04-06 20:12:02] s05n07 now marked for drain: sanity-check: not enough space on /scratch - less than 100G 
[2020-04-06 20:10:02] s05n07 resumed
[2020-04-06 19:52:02] s05n01 resumed
[2020-04-06 19:48:02] s05n06 now marked for drain: sanity-check: not enough space on /scratch - less than 100G 
[2020-04-06 19:48:02] s05n05 now marked for drain: sanity-check: not enough space on /scratch - less than 100G 
[2020-04-06 19:46:02] s05n05 resumed
[2020-04-06 19:32:03] s05n13 now marked for drain: sanity-check: not enough space on /scratch - less than 100G 
[2020-04-06 19:32:03] s05n06 resumed
[2020-04-06 19:30:04] s05n13 resumed
[2020-04-06 18:58:03] s05n06 now marked for drain: sanity-check: not enough space on /scratch - less than 100G 
[2020-04-06 18:54:02] s05n14 resumed
[2020-04-06 18:46:02] s05n06 resumed
[2020-04-06 18:18:02] s05n06 now marked for drain: sanity-check: not enough space on /scratch - less than 100G 
[2020-04-06 18:14:04] s05n07 now marked for drain: sanity-check: not enough space on /scratch - less than 100G 
[2020-04-06 18:12:03] s05n08 resumed
[2020-04-06 18:10:02] s05n07 resumed
[2020-04-06 18:10:02] s05n06 resumed
[2020-04-06 18:04:02] s05n05 now marked for drain: sanity-check: not enough space on /scratch - less than 100G 
[2020-04-06 18:04:02] s05n03 resumed
[2020-04-06 17:54:02] s05n07 now marked for drain: sanity-check: not enough space on /scratch - less than 100G 
[2020-04-06 17:44:02] s05n03 now marked for drain: sanity-check: not enough space on /scratch - less than 100G 
[2020-04-06 17:42:03] s05n05 resumed
[2020-04-06 17:42:03] s05n02 resumed
[2020-04-06 17:32:02] s05n10 resumed
[2020-04-06 17:30:03] s05n05 now marked for drain: sanity-check: not enough space on /scratch - less than 100G 
[2020-04-06 17:28:02] s05n05 resumed
[2020-04-06 17:28:02] s05n03 resumed
[2020-04-06 17:18:02] s05n02 now marked for drain: sanity-check: not enough space on /scratch - less than 100G 
[2020-04-06 17:16:02] s05n12 resumed
[2020-04-06 17:16:02] s05n04 resumed
[2020-04-06 17:16:02] s05n01 now marked for drain: sanity-check: not enough space on /scratch - less than 100G 
[2020-04-06 17:10:03] s05n10 now marked for drain: sanity-check: not enough space on /scratch - less than 100G 
[2020-04-06 17:10:03] s05n08 now marked for drain: sanity-check: not enough space on /scratch - less than 100G 
[2020-04-06 17:10:03] s05n03 now marked for drain: sanity-check: not enough space on /scratch - less than 100G 
[2020-04-06 17:08:03] s05n10 resumed
[2020-04-06 17:06:02] s05n06 now marked for drain: sanity-check: not enough space on /scratch - less than 100G 
[2020-04-08 05:00:02]
Backup on    fs1  finished 0-03:16:34 ago
Backup on    fs2 initiated 6-11:15:00 ago
Backup on    fs3 initiated 0-03:00:00 ago
Backup on    fs5 initiated 2-07:00:01 ago
Backup on    fs6  finished 1-23:21:59 ago
Backup on s95n01  finished 3-07:06:57 ago
[2020-04-08 05:36:05]
################
# RUNNING JOBS #
################
Count  USER      MIN_CPUS  MIN_MEMORY  TIMELIMIT    QOS
-----  ----      --------  ----------  ---------    ---
1      ijjduan   24        500G        4-00:00:00   qos1
1      lucie     1         110G        2-22:00:00   bumped
1      michaelk  36        128G        12-00:00:00  gold
1      mtxellrb  6         300G        12:00:00     bumped
1      owyn      24        1T          8-12:00:00   basic
1      samuele   10        250G        1-00:00:00   bumped
1      samuele   10        250G        1-00:00:00   qos1
1      samuele   12        200G        12:00:00     qos1
2      andyb     16        8G          2:08:00      qos1
2      fprive    16        128G        12-00:00:00  qos1
2      mtv       16        64G         3-00:00:00   bumped
4      anbt      6         36G         12:00:00     bumped
14     clara     24        256G        3-00:00:00   qos1
14     igrove    1         14G         16-16:00:00  qos2
22     kmt       1         16G         5-00:00:00   bumped
24     agomez    1         4G          12:00:00     bumped
73     andyb     16        128M        4:16:00      qos1
100    lucie     1         110G        2-22:00:00   qos1
125    juraj     8         8G          12:00:00     basic
198    agomez    1         4G          12:00:00     qos1
###############
# QUEUED JOBS #
###############
Count  USER     MIN_CPUS  MIN_MEMORY  TIMELIMIT    QOS
-----  ----     --------  ----------  ---------    ---
1      akiratk  256       8G          11:50:00     basic
1      andyb    1         16G         1:00:00      qos1
1      andyb    1         512M        8:00         qos1
1      clara    36        384G        3-00:00:00   qos1
1      ials     35        64G         2-00:00:00   bumped
1      lucie    1         100G        20:00:00     qos1
2      mtv      16        64G         3-00:00:00   basic
3      andyb    16        4G          4:16:00      qos1
9      ials     35        64G         2-00:00:00   qos1
14     andyb    1         16G         8:00         qos1
14     lucie    24        256G        12:00:00     qos1
17     andyb    1         128M        8:00         qos1
49     juraj    2         8G          16:00:00     basic
53     clara    24        256G        3-00:00:00   qos1
112    fprive   16        128G        12-00:00:00  qos1
218    anbt     6         36G         12:00:00     basic
313    andyb    16        8G          2:08:00      qos1
500    andyb    16        128M        4:16:00      qos1
1099   andyb    1         128M        4:00         qos1
1410   lucie    1         110G        2-22:00:00   qos1
2761   agomez   1         4G          12:00:00     qos1
8831   juraj    8         8G          12:00:00     basic
 
#################
# CPUs per User #
#################
NCPUs  USER      QOS
-----  ----      ---
1      lucie     bumped
6      mtxellrb  bumped
10     samuele   bumped
14     igrove    qos2
22     kmt       bumped
22     samuele   qos1
24     agomez    bumped
24     anbt      bumped
24     ijjduan   qos1
24     owyn      basic
32     fprive    qos1
32     mtv       bumped
36     michaelk  gold
100    lucie     qos1
198    agomez    qos1
336    clara     qos1
1000   juraj     basic
1200   andyb     qos1
########################
# QUEUED CPUs per User #
########################
NCPUs  USER     QOS
-----  ----     ---
32     mtv      basic
35     ials     bumped
256    akiratk  basic
315    ials     qos1
1308   anbt     basic
1308   clara    qos1
1747   lucie    qos1
1792   fprive   qos1
2761   agomez   qos1
14188  andyb    qos1
70746  juraj    basic