System down - What would you do?

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

key

New Member
Sep 6, 2018
4
0
1
Hello Expert,

Our system just get hit by an unauthorized access which bring down the HP SAN storage.

What would you do to bring up the storage ASAP

Here is the background of our system:

  • Two HP physical servers with Win 2008R2 running SAP application. The OS is on local C drive while SAP application is pointing to HP SAN storage.
  • Another HP physical server that running Cognos data warehouse with the data pointing HP SAN storage
  • All data on SAN storage was less than 1TB of data
  • The 3 servers and the SAN are less than 4 years old.
  • The servers, network and Cognos was designed and build by outside consultants.
  • The SAP application was build and configure by outside developer.
  • Teamviewer was used by IT members, users and customer for remote support
Backup status:

  • HP 24 tapes library.
  • Netgear Ready NAS 2100
  • Backup are running everyday to the disk and to the tapes.
  • Backup Tapes will be sent out to the vault every morning.
  • Twice a year, an IT administrator performed a system restore the SAP servers from the backup to the test environment and have users tested and validated random program to ensure all figures are match between live system and restored system.
Here is the issue:

  • An unauthorized access from former IT administrator access the network through Teamviewer and deleted HP SAN configuration. This deletion put the SAN storage back to factory default setting which clear out all the data on the storage
  • He also deleted the backup from the disk.
  • There is no damage to 3 physical OS server.
All the experts out there, please chime in and let me know what would you do to bring the server up and running ASAP at the shortest time?

Will "delete SAN configuration" or "delete backup data from NAS device cause hardware failure?
 

Patrick

Administrator
Staff member
Dec 21, 2010
12,511
5,792
113
@markpower28 has the right idea

Also, never use TeamViewer. Beyond ex-admins, there are groups constantly trying to break into machines accessible via these remote access apps.
 
  • Like
Reactions: Tha_14 and William

zedascuras

New Member
Feb 15, 2015
12
1
3
39
As it was said above, recover from tape, bring the last two days of backup and restore the latest day, if anything goes wrong, then try the two days old backup.
Call HP support to rebuild the SAN config.
Accept the lose of today's data and disable team viewer as a remote access tool on the production servers.
 

cesmith9999

Well-Known Member
Mar 26, 2013
1,417
468
83
(Side note) I hope that you get the person that did this sabotage held up on criminal charges. Then civil suit to recoup your downtime costs. This is a separate issue that your legal team can start.

Which HP SAN? Depending on which model HP may be able to rebuild the server from configuration backups.

were any of your past practice restores successful? are you still friends with the original consultants? are they able to help with the restore of the configuration?



Chris
 

Evan

Well-Known Member
Jan 6, 2016
3,346
598
113
What part of the world are you in ? Maybe I could point you in the direction of a trusted expert to help myself.
Of less importance but to find the right person are we talking a HANA, Oracle, or MS SQL DB ?
 

Connorise

Member
Mar 2, 2017
75
17
8
33
US. Cambridge
"I hope that you get the person that did this sabotage held up on criminal charges" Agree, these actions should be punished.

Just to add, you might want to store one copy of your backups in the cloud, exactly for situations when the whole site is compromised.
 

EffrafaxOfWug

Radioactive Member
Feb 12, 2015
1,394
511
113
A backup you can't recover from when the entire site is compromised isn't much of a backup at all and keeping online copies in the cloud won't help you if you infrastructure is compromised at that level (at least unless if there's some "don't ever delete this stuff even if I tell you to" switch built into the program, something I've never seen personally). If you can afford it, keep some tapes or HDDs offline in a safe or a vault somewhere.

I've got to wonder how this guy got in in the first place though...? Did no-one think to disable their user account? And how did access to the server get them any sort of access to the SAN management interface?

Back on topic, if the entirety of the data on the SAN was <1TB, I'm assuming it's only a subset of a SAN that was compromised and not the whole thing...? And what's the role of the ReadyNAS, just there as a warm standby/VTL?
 

key

New Member
Sep 6, 2018
4
0
1
Hi Effrafax,
It's just a small SAN (HP Eva 4400 storagework) and the configuration got clear out which put the SAN back to manufacture default setting that caused all the data lost.
The readyNAS is just a backup storage for Symantec backup.
 

WANg

Well-Known Member
Jun 10, 2018
1,302
967
113
46
New York, NY
What's the alternative?
Literally anything else - or rather, the remote management utility should not be able to traverse NATs, ignore network topography and be a single source of failure.
I mean, at the very least there should be at least one more layer to compromise, mandating the need to VPN in, get through a jump box, get an RDP session from a recognized IP, stuff like that. It's not really putting your eggs in one very strong basket as much as throwing up multiple layers of defense and giving you enough time and intrusion alerts to react to it.
 

WANg

Well-Known Member
Jun 10, 2018
1,302
967
113
46
New York, NY
Hello Expert,

Our system just get hit by an unauthorized access which bring down the HP SAN storage.

What would you do to bring up the storage ASAP

Here is the background of our system:

  • Two HP physical servers with Win 2008R2 running SAP application. The OS is on local C drive while SAP application is pointing to HP SAN storage.
  • Another HP physical server that running Cognos data warehouse with the data pointing HP SAN storage
  • All data on SAN storage was less than 1TB of data
  • The 3 servers and the SAN are less than 4 years old.
  • The servers, network and Cognos was designed and build by outside consultants.
  • The SAP application was build and configure by outside developer.
  • Teamviewer was used by IT members, users and customer for remote support
Backup status:

  • HP 24 tapes library.
  • Netgear Ready NAS 2100
  • Backup are running everyday to the disk and to the tapes.
  • Backup Tapes will be sent out to the vault every morning.
  • Twice a year, an IT administrator performed a system restore the SAP servers from the backup to the test environment and have users tested and validated random program to ensure all figures are match between live system and restored system.
Here is the issue:

  • An unauthorized access from former IT administrator access the network through Teamviewer and deleted HP SAN configuration. This deletion put the SAN storage back to factory default setting which clear out all the data on the storage
  • He also deleted the backup from the disk.
  • There is no damage to 3 physical OS server.
All the experts out there, please chime in and let me know what would you do to bring the server up and running ASAP at the shortest time?

Will "delete SAN configuration" or "delete backup data from NAS device cause hardware failure?
Fastest way to get things back? Throw money at the problem. If you send a check to HPe for support, call them. If the consultants are on retainer, also call them. Call everyone who has ever touched the system and seek their assistance.

You really have 2 tasks here right now
- Resumption of services
- Gathering information to prosecute the intruder

There's criminality involved here, and someone in the C-level suite will no doubt be in the server room looking for answers. What you need now is answer not "can we return back to where things were ASAP" as much as "holy crap, what the hell else did he do before this?"

If this A-hole is audacious enough to jump in on Teamviewer and kill the SAN configuration (assuming that you have the logs to provide that it was indeed the culprit), I am sure that he/she probably did other stuff as well, so who knows. Maybe the daily backup tape contents are encrypted and when he/she left, he pulled the keys out and flushed it.
 

Myth

Member
Feb 27, 2018
148
7
18
Los Angeles
This is really a big fear of mine as well. We have a previous employee who knows a lot of old passwords. We tried to change as many as possible, but I fear that if he goes self-destruct he will get in and wipe client data just to lower our reputation.

He actually was in there via teamviewer once before, on a client machine, the client invited him in, but still scared the crap out of me. He left a file transfer window open on accident of some log files that he captured from the server. I was able to screenshot them and forward that data to the lawyer.

We changed the teamviewer password, but it doesn't keep logs of who comes in and who goes. At least I can't find them, it's not very secure. I know I shouldn't use it, but man it's really convenient. The system logs of the authentication also don't show who is loging in and from where. It will just say logon successful or not.

I imagine we would need some sort of intrusion detection system setup on each client network to monitor activity like this and keep logs. I don't know how else, does anyone know how to keep teamviewer logs and show who's IP is connecting to us?

It's just a big mess.
 

LaMerk

Member
Jun 13, 2017
38
7
8
33
As far as I know, TeamViewer keep information about the connections in C:\Program Files (x86)\TeamViewer\Connections_incoming.txt file, so you can check it.
 
  • Like
Reactions: Tha_14

Tha_14

Server Newbie
Mar 9, 2017
72
10
8
As far as I know, TeamViewer keep information about the connections in C:\Program Files (x86)\TeamViewer\Connections_incoming.txt file, so you can check it.
@OP: Knowing the IP might not be much help if they know what they are doing but make sure to check the teamviewer logs eitger way. Hopefully the guy behind this is not smart.
 
  • Like
Reactions: dswartz