It stopped again, and this is the console output. It says something about not enough space, but it should be plenty.
Code:
Filesystem Size Used Avail Use% Mounted on
rpool/ROOT/napp-it-0.9c1 13G 2.2G 11G 18% /
Console output:
Code:
rm: cannot remove '/tmp/nappit/*.htm': No such file or directory
rm: cannot remove '/tmp/nappit/socket-basic-stat_minute.*': No such file or directory
Content-type: text/html
<html>
<head>
<title>lib read_ahfile (taskexec.pl 36) file not found</title>
</head>
<body>
<tt>lib read_ahfile (taskexec.pl 36) file not found</tt>
<br><hr><tt><b>Fragen Sie bei Bedarf Ihren Systembetreuer</b></tt>
lib read_ahfile (taskexec.pl 36) file not found at /var/web-gui/data/wwwroot/cgi-bin/admin-lib.pl line 423.
Content-type: text/html
<html>
<head>
<title>lib read_ahfile (taskexec.pl 36) file not found</title>
</head>
<body>
<tt>lib read_ahfile (taskexec.pl 36) file not found</tt>
<br><hr><tt><b>Fragen Sie bei Bedarf Ihren Systembetreuer</b></tt>
lib read_ahfile (taskexec.pl 36) file not found at /var/web-gui/data/wwwroot/cgi-bin/admin-lib.pl line 423.
[Sat Jul 6 01:51:14 2013] [debug] GET /echo (Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.116 Safari/537.36).
[Sat Jul 6 01:51:14 2013] [debug] Routing to a callback.
[Sat Jul 6 01:51:14 2013] [debug] 101 Switching Protocols (0.000565s, 1769.912/s).
Content-type: text/html
<html>
<head>
<title>lib read_ahfile (taskexec.pl 36) file not found</title>
</head>
<body>
<tt>lib read_ahfile (taskexec.pl 36) file not found</tt>
<br><hr><tt><b>Fragen Sie bei Bedarf Ihren Systembetreuer</b></tt>
lib read_ahfile (taskexec.pl 36) file not found at /var/web-gui/data/wwwroot/cgi-bin/admin-lib.pl line 423.
kill: 23621: No such process
[Sat Jul 6 02:25:13 2013] [debug] GET /echo (Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.116 Safari/537.36).
[Sat Jul 6 02:25:13 2013] [debug] Routing to a callback.
[Sat Jul 6 02:25:13 2013] [debug] 101 Switching Protocols (0.000470s, 2127.660/s).
/var/web-gui/data/napp-it/zfsos/_lib/scripts/taskserver.sh: line 17: 23621 Killed nice /usr/bin/perl /var/web-gui/data/napp-it/zfsos/_lib/scripts/taskexec.pl $F
Content-type: text/html
<html>
<head>
<title>lib read_ahfile (taskexec.pl 36) file not found</title>
</head>
<body>
<tt>lib read_ahfile (taskexec.pl 36) file not found</tt>
<br><hr><tt><b>Fragen Sie bei Bedarf Ihren Systembetreuer</b></tt>
lib read_ahfile (taskexec.pl 36) file not found at /var/web-gui/data/wwwroot/cgi-bin/admin-lib.pl line 423.
Content-type: text/html
<html>
<head>
<title>lib read_ahfile (taskexec.pl 36) readerror</title>
</head>
<body>
<tt>lib read_ahfile (taskexec.pl 36) readerror</tt>
<br><hr><tt><b>Fragen Sie bei Bedarf Ihren Systembetreuer</b></tt>
lib read_ahfile (taskexec.pl 36) readerror at /var/web-gui/data/wwwroot/cgi-bin/admin-lib.pl line 423.
/var/web-gui/data/napp-it/zfsos/_lib/scripts/taskserver.sh: fork: Not enough space
/var/web-gui/data/napp-it/zfsos/_lib/scripts/taskserver.sh: fork: Not enough space
jk@xxxxx:~$ [Sat Jul 6 13:03:12 2013] [debug] GET /echo (Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.116 Safari/537.36).
[Sat Jul 6 13:03:12 2013] [debug] Routing to a callback.
[Sat Jul 6 13:03:12 2013] [debug] 101 Switching Protocols (0.000327s, 3058.104/s).
Last output from messages:
Code:
Jul 6 09:32:58 hostname tmpfs: [ID 518458 kern.warning] WARNING: /etc/svc/volatile: File system full, swap space limit exceeded
As far as I can tell swap space looks fine:
Code:
df -h | grep swap
swap 1.7G 304K 1.7G 1% /etc/svc/volatile
swap 1.7G 40K 1.7G 1% /tmp
swap 1.7G 52K 1.7G 1% /var/run
Code:
swapfile dev swaplo blocks free
/dev/zvol/dsk/rpool/swap 92,2 8 2097144 2048304
Any advice?
EDIT:
Crashed again. Same error "/var/web-gui/data/napp-it/zfsos/_lib/scripts/taskserver.sh: fork: Not enough space".
As far as I can tell, that means I have runned out of memory. OmniOS VM has 6gb dedicated, and no limit on how much memory of the total (24gb) it can use through ESXi.
Should I do something different?
EDIT2:
I've tried to increase swap space too see if that helps. We'll see. Any other advice is appreciated.