StarWind VSA Linux Version (sort of)

Bjorn Smith

Active Member
Sep 3, 2019
207
65
28
Hi,

I looked into StarWind VSA, but discarded the idea since it required a windows machine to run it.

Then today I found out that there is a linux version of it - great I thought - so I proceeded to download the VSA and installed it in vsphere.

I was poking around on the linux machine that it runs on and guess what ps aux showed me:
Code:
root      2020  0.0  0.0 127872  1268 ?        Ss   22:05   0:00 SCREEN -dmS StarWindVSA /bin/bash -c WINEDEBUG=-all WINEPREFIX=/opt/StarWind/StarWindVSA /usr/bin/wine64 StarWindService.exe --console -o -lpStarWindVSA
root 2022 0.0 0.0 115408 1476 ? S 22:05 0:00 /bin/bash /usr/bin/logwatcher /var/StarWind/StarWindVSA/events/StarWindVSA-Events.log
root 2023 0.6 0.3 2213912 30596 pts/0 Ssl+ 22:05 0:00 StarWindService.exe --console -o -lpStarWindVSA
root 2029 0.8 0.0 17760 6088 ? Ss 22:05 0:00 /usr/bin/wineserver
root 2035 0.0 0.0 2069596 3424 ? Ssl 22:05 0:00 C:\windows\system32\services.exe
root 2038 0.0 0.0 2055580 5528 ? Sl 22:05 0:00 C:\windows\system32\winedevice.exe
root 2040 0.0 0.0 1920280 7132 pts/0 Sl+ 22:05 0:00 C:\windows\system32\explorer.exe /desktop
root 2047 0.2 0.0 451592 6820 ? Ssl 22:05 0:00 /usr/libexec/udisks2/udisksd
root 2054 0.0 0.0 1938272 2924 ? Sl 22:05 0:00 C:\windows\system32\plugplay.exe
root 2062 0.3 0.0 2043496 5756 ? Sl 22:05 0:00 C:\windows\system32\winedevice.exe
root 2146 0.0 0.0 6524 652 ? S 22:05 0:00 inotifywait -q --format %e /var/StarWind/StarWindVSA/events/StarWindVSA-Events.log
root 2147 0.0 0.0 115408 404 ? S 22:05 0:00 /bin/bash /usr/bin/logwatcher /var/StarWind/StarWindVSA/events/StarWindVSA-Events.log
root 2157 0.1 0.0 163596 6192 ? Ss 22:05 0:00 sshd: admin [priv]
Suffice to say I was surprised :)

Lets just say I will not be running their Linux version after all - I mean virtualized storage is one thing, but virtualized storage running in a virtualized layer on top of a VM, that is just crazy.

Perhaps one day if they create a real linux binary I will take a look at it again.

I don't know if this is common knowlede, but I certainly did not know they were running their windows binary on Wine :)
 

RTM

Active Member
Jan 26, 2014
520
185
43
To be fair, it looks like they are using Wine (Wine Is Not an Emulator), which is (also) not nested virtualization.
That all said, using Wine in a server environment is very unorthodox, certainly not something I would ever consider using in production.

It would be interesting to see what the impact of it is, I suppose it is possible that they are doing it with something that isn't part of anything critical, and a result of a crash is perfectly acceptable.
 

Bjorn Smith

Active Member
Sep 3, 2019
207
65
28
I suppose it is possible that they are doing it with something that isn't part of anything critical
Did you look at the process list?

They are running the actualy starwind server windows executable under Wine -
Code:
root      2020  0.0  0.0 127872  1268 ?        Ss   22:05   0:00 SCREEN -dmS StarWindVSA /bin/bash -c WINEDEBUG=-all WINEPREFIX=/opt/StarWind/StarWindVSA /usr/bin/wine64 StarWindService.exe --console -o -lpStarWindVSA
It might not be nested virtualization, but its certainly not a vanilla VM - and Wine might not be a virtualization layer - but its not straight to kernel calls like it would have been if it was running on windows - or if it had been a real linux binary.
 

RTM

Active Member
Jan 26, 2014
520
185
43
Yeah... I did see it, and again I would argue that it is (at least theoretically) possible that it is isn't doing absolutely critical (though I doubt it).

I suppose what I am getting at, is that I would like to have been a fly on the wall in the meeting, where they decided this was an acceptable way of going forward, to get a glimpse of what thoughts went through their minds :)

Looking closer it is also rather peculiar that they are using "screen" to run their service and background it rather than Systemd.
Looking even closer, it also appears to be running as root, something that might actually get classified as a security vulnerability :eek:
 
Last edited:
  • Like
Reactions: Taras Shved

Bjorn Smith

Active Member
Sep 3, 2019
207
65
28
I suppose what I am getting at, is that I would like to have been a fly on the wall in the meeting
Yeah me too - my guess is that either they thought nobody would find out - since the first versions of the Linux VSA did not have ssh open or allowed logins, so it would not have been easy to find out - or they really think it is an okay solution, or the CEO or whatever put his foot down and said - "I want a linux version _NOW_" - and it kind of is a linux version - I mean I have run windows game servers on wine and it runs okay - but I would never trust my data with it.

rather peculiar that they are using "screen
I think it is because some windows programs cannot run without a console available - I think it is the way they are written - as far as i know even windows services has a console available.

Furthermore I have sent a mail to StarWind asking if they plan on making a real linux version availalbe :) Lets see what their response will be.
 

Taras Shved

New Member
May 22, 2020
2
1
3
www.starwindsoftware.com
Hello, Bjorn, RTM, and thank you so much for your keen interest and curiosity regarding our product and for sharing your thoughts and vision with the Community. We, here at StarWind, appreciate that so much.

Same code
We are developing our flagship product, maintaining the same branch of code for both use cases Windows and Linux. This approach makes sure that customers using each of the flavors get the same top-level user-experience in terms of improvements, bug fixes, and new features. However, all the performance-critical storage- and OS-related API calls are made natively, depending on the environment where the product is currently used. You can easily check this by benchmarking the storage performance on our Linux VSA, which is mostly similar or even better compared to the Windows version, depending on the particular storage configuration and workload. Needless to say that this would be impossible if bottlenecked by a single Wine service instance. We are still using Wine for compatibility purposes because some management plugins related to init and start of the service within our code require WIN32 API calls. Yet, you are not the first who gets confused with Wine presence, and we are continually working towards removing Wine binaries from our Linux VSA mainly to reduce the footprint and resources usage rather than because of other reasons.

Root
Linux VSA is always running on top of a hypervisor in an isolated environment. We do not consider using root for system services as an issue as long as the root password is secured. But I much appreciate your feedback, and we will surely reconsider this approach and change it if necessary within the next release already.

Systemd
Due to StarWind service's current architecture, the "screen" mode is needed to run it in the background. It allows service to handle status commands (start, stop, status) properly. Service itself always runs without the console. Systemd service itself is generated by systemd-sysv-generator, which creates wrapper .service units for SysV init scripts in /etc/init.d/* at boot and when the configuration of the system manager is reloaded. This allows systemd to support them similarly to native units.

We want Linux version *NOW*
It took us almost a year and hundreds of hours of testing to bring the Linux version to you, which would correspond to the quality and reliability to which our users of Windows version are used to. Nowadays, we have hundreds of customers running our Linux VSA mainly on vSphere but also on other hypervisors in hyperconverged and converged scenarios successfully.

We are open
We do not intentionally prevent our users and customers from getting any information about our products and technologies. Our technical staff is entirely open to any sorts of questions and do their best to answer all of them as full as possible. Those of you, who have worked with our support team, be it Solution Architects or Technical Support Engineers, could surely notice that. That is also one of the reasons StarWind got the Gartner Peer Insights “Voice of the Customer” Hyperconverged Infrastructure Customers’ Choice award https://www.gartner.com/doc/reprints?id=1-1YZ70V1W&ct=200506&st=sb.

If you would like to discuss any further technical details regarding our product, see our test lab, or show off yours, I will do my best to connect you with the Linux VSA product owner directly. Your feedback is highly valued, and we will be more than happy to discuss these questions directly if you wish. I will make sure your account manager introduces you directly to PM ASAP.

Thanks once again and hope to talk to you soon.
 

Bjorn Smith

Active Member
Sep 3, 2019
207
65
28
Hi,
@Taras Shved

I am sorry if my post was understood as if I was saying your software was bad - it was just a post to show my surprise - everyone jumps through hoops in software development - I have just never seen this combination.

However, all the performance-critical storage- and OS-related API calls are made natively
I don't see how that is possible when you run a windows binary through wine and I did not see any other starwind processes running - just wine with the windows executable.

I know wine translates windows win32 api calls to linux calls, but its still a translation - not natively - unless of course you somehow have made a custom version of wine that is optimised to your binary.

storage performance on our Linux VSA, which is mostly similar or even better compared to the Windows version
I don't doubt your benchmarks show mostly same performance - I am sure you would not have launched it if you had worse performance.

And it is an achievement in itself to be able to run a windows binary in wine - and in particular one like starwind.

It is possible to cross compile the same code base to different operating systems, so I still don't understand why you went the wine way instead of working towards compiling the same codebase to linux.

I mean it is "fairly" standard what your software does - in the sense that it should be possible both on windows and linux - it would be different if you were building a COM service, since COM is a windows only technology - but storage, network, service etc are built everywhere. I am in no way trying to dimish the features/performance or anything else of starwind vsa - but what it does is possible to get working natively on linux as well - except of course deep windows integration, like eventlog etc, since those are windows only concepts.

I am in no way saying it would be easy to port your software to linux as a native product, but it is certainly possible.
 

RTM

Active Member
Jan 26, 2014
520
185
43
First of all kudos, for actually replying to a post like this on a "random" forum:)
Root
Linux VSA is always running on top of a hypervisor in an isolated environment. We do not consider using root for system services as an issue as long as the root password is secured. But I much appreciate your feedback, and we will surely reconsider this approach and change it if necessary within the next release already.
I have no experience with your overall architecture, but the problem with running stuff as root is not so much a question about the password.
The problem is if someone is able to compromise the management application itself to do code execution (I assume it has a webinterface? that is a common vector), those commands will be run as root, essentially resulting in a full system compromise very quickly.

I get that a user can defend themself somewhat, by doing proper network segmentation etc., but that is hardly sufficient and many people/companies will fail to do this or fail to do it properly. This is why you do stuff like the principle of least privilege and defense in depth because security controls (like network segmentation) fails.

A better way (though it will probably require some redoing on your end) is to run the service with a fairly unprivileged user, and have that user access what requires higher privileges via sudo (with sudo you can define which commands a user is allowed to execute, ideally you do that as closely as what needs to have, meaning it is better to make a script that does exactly what needs to be done, and give the user access to run that via sudo, than it is to give the user access to the general tools).

This is just one way of doing it of course, there are many ways to help you drop privileges.
 
  • Like
Reactions: NISMO1968

NISMO1968

[ ... ]
Oct 19, 2013
78
13
8
San Antonio, TX
www.vmware.com
You isolate performance crucial primitives (like say mutexes and critical sections, each one non-acquired right on call is putting calling thread into APC state so you get at least scheduler timeout which is ~30ms, done with WINE you'll get even more thread context switches and re-queueing so it's SLOW) into Linux-native shared libs, you compile your Windows application with wine.lib and these shared libs (Yes, it WON'T RUN on Windows anymore!) and... voila! You save yourself man-years of the software engineering efforts, you have unified codebase (more or less...) and your code runs on Windows and Linux and performance close / the same.


"What you gain by recompiling your application with Winelib is the ability to make calls to Unix APIs, directly from your Windows source code. This allows for a better integration with the Unix environment than is allowed by running an unmodified Windows application running in Wine."

A propos, there are NO other processed except your binary running and seen with top or whatever!

Hi,
@Taras Shved

[ ... ]

I know wine translates windows win32 api calls to linux calls, but its still a translation - not natively - unless of course you somehow have made a custom version of wine that is optimised to your binary.



[ ... ]
 

NISMO1968

[ ... ]
Oct 19, 2013
78
13
8
San Antonio, TX
www.vmware.com
This is very naive :) I wish software engineering would happen in your Universe where unicorns eat rainbow and dump with butterflies... In our Universe environment is much more harsh :( In a nutshell: Properly written system software will benefit from all the architectural features particular operating system has. Say on Windows you want to use I/O Completion Ports with your block devices to achieve maximum performance and lowest latency response, I/O Completion Ports are completely missing from Linux... Doing RDMA to eliminate as much network latency and CPU usage as possible requires Network Direct Kernel Provider Interface (NDKPI) on Windows (R.I.P. Don Stanwyck) and so-called "verbs" on Linux, and it's not only these APIs are different, but the whole concept they represent has virtually nothing in common! Making long story short: Yes, you can port your "Hello world!" style and completely level app from one OS to another, but it's not possible for most of the commercial software you'll see on the market.

P.S. There's an exception, software written with portability in mind has something called "Software Abstraction Layer", but closer you get to the hardware to squeeze maximum performance - more complex this level translating virtual API calls to system-dependent ones will be. + like I've told there's no translation really for hardcore stuff, so "translate" means "re-implement"...

Hi,
@Taras Shved

[ ... ]

It is possible to cross compile the same code base to different operating systems, so I still don't understand why you went the wine way instead of working towards compiling the same codebase to linux.

I mean it is "fairly" standard what your software does - in the sense that it should be possible both on windows and linux - it would be different if you were building a COM service, since COM is a windows only technology - but storage, network, service etc are built everywhere. I am in no way trying to dimish the features/performance or anything else of starwind vsa - but what it does is possible to get working natively on linux as well - except of course deep windows integration, like eventlog etc, since those are windows only concepts.

I am in no way saying it would be easy to port your software to linux as a native product, but it is certainly possible.
 

NISMO1968

[ ... ]
Oct 19, 2013
78
13
8
San Antonio, TX
www.vmware.com
Man you have a point! Less software runs @ escalated privilege level - less grey hairs admins will develop while running it. I'd be very careful (read - concerned) with storage stack having "root" rights, even with VM-level isolation.

First of all kudos, for actually replying to a post like this on a "random" forum:)

I have no experience with your overall architecture, but the problem with running stuff as root is not so much a question about the password.
The problem is if someone is able to compromise the management application itself to do code execution (I assume it has a webinterface? that is a common vector), those commands will be run as root, essentially resulting in a full system compromise very quickly.

I get that a user can defend themself somewhat, by doing proper network segmentation etc., but that is hardly sufficient and many people/companies will fail to do this or fail to do it properly. This is why you do stuff like the principle of least privilege and defense in depth because security controls (like network segmentation) fails.

A better way (though it will probably require some redoing on your end) is to run the service with a fairly unprivileged user, and have that user access what requires higher privileges via sudo (with sudo you can define which commands a user is allowed to execute, ideally you do that as closely as what needs to have, meaning it is better to make a script that does exactly what needs to be done, and give the user access to run that via sudo, than it is to give the user access to the general tools).

This is just one way of doing it of course, there are many ways to help you drop privileges.
 

Bjorn Smith

Active Member
Sep 3, 2019
207
65
28
with wine.lib
Ok, I was not aware that wine had actually released a static win32 library that you could compile against to replace the windows (ms) static library, that is neat.

This is very naive
Ok, no need to get harsh - I know how software works - I have been building software for a living since the mid 90's - my point still stands - if StarWind wanted to do this right, they would rewrite their core using a framework that can be compiled on both platforms - i.e. start by using only std c++ or e.g. boost which makes cross compiling easier.

Then they could have their windows specific code in one place, linux specific code in another and the shared stuff in portable c++.

But of course I also know how the world works. You start out by building your software, slowly painting yourself into a corner, because you want to squeeze every bit of performance out of your code and suddenly you end up in a place where your code cannot be ported anywhere because you use operating system primitives because its faster.

Then after a while your boss comes around and says, "oh btw, we would like for our software to run on this platform - that is not problem is it?"
And being the good little soldier you are, you respond "No sir, not at all - it just requires a little testing/tweaking" - when it reality you die a little inside because you remember all the operating system specific hacks you have made to make it run 0.05% faster.
 

Taras Shved

New Member
May 22, 2020
2
1
3
www.starwindsoftware.com
I am sorry if my post was understood as if I was saying your software was bad - it was just a post to show my surprise - everyone jumps through hoops in software development - I have just never seen this combination.
Thank you so much for such an intense discussion. It is a pleasant surprise and incredibly motivating to face such an in-depth technical and exciting analysis on a "random" forum.

Man you have a point! Less software runs @ escalated privilege level - less grey hairs admins will develop while running it. I'd be very careful (read - concerned) with storage stack having "root" rights, even with VM-level isolation.
As promised previously, the de-escalation of the running code inside the VSA is already in our feature request list for the upcoming release.

Ok, I was not aware that wine had actually released a static win32 library that you could compile against to replace the windows (ms) static library, that is neat.
Also, I would like to mention that even 0.05% of overall performance may result in hundreds of thousands of IOps if working with cutting-edge hardware True Record Setting Hyperconverged Performance | Mellanox Technologies Blog.
 
  • Like
Reactions: RTM