I recently was working on a larger ETL process that started with the reception of various data files via SFTP that were delivered on varying schedules. The requirement was that as files are received we generate a unique event in a database, then execute a sequence of commands to archive the files out of the delivery directory and offline to a central immutable annotated file repository.
This new functionality had to integrate with an existing SFTP legacy server, and would likely have other uses outside of this initial use-case.
Looking around for simple solutions based on a scripting language, I really could not find any that would work or be extensible enough for the need. Hence I ended up writing io-event-reactor.
The basic concept is this; you have a monitor that listens for IO events for particular paths on the filesystem. As these IO events occur, they are passed on to one or more evaluators to decide whether or not the IoEvent should be reacted to by one or more configured reactors. The entire latter sequence is encapsulated in an IoReactor instance that manages the flow between the three described components.
With this module, you construct and configure a single IoReactorService which can manage and contain one or more IoReactor instances, as many as you wish, providing for lots of flexibility for reacting to filesystem events.
When you configure the IoReactorService and its IoReactor instances, you specify which plugins you would like to use to fulfill the monitor and reactor roles. For evaluators you simply provide one or more functions which evaluate whether or not an IoEvent should be passed on to one or more reactors.
For reactor plugins, I developed two based on my initial needs.
- One for inserting records into a MySql database via node-mysql, available at: https://github.com/bitsofinfo/io-event-reactor-plugin-mysql
- The other for executing shell commands based on stateful-process-command-proxy, available at: https://github.com/bitsofinfo/io-event-reactor-plugin-shell-exec
For an real-world example of the kind of application you could build on top of this, check out io-overwatch (albiet a simple utility) at: https://github.com/bitsofinfo/io-overwatch
Building on my last post on stateful-process-command-executor this post will cover how you can leverage that node.js module to expose the capabilities of Powershell cmdlets over a REST api presented through an AngularJS interface. Why would one want to do this you ask? Well I’ve covered this in my last post but I will briefly explain it here.
(Note, what is described below could just as easily be built for Bash processes as well as the underlying module supports it)
The use case came out of the need to automate certain calls to manage various objects within Microsoft o365’s environment. Unfortunately Microsoft’s GraphAPI, does not expose all of the functionality that is available via its suite of various Powershell cmdlets for o365 services. Secondly when you need to do these operations via Powershell, its requires a per-established remote PSSession to o365…. and establishing (and tearing down) a new remote PSSession whenever you need to invoke a cmdlet against a remote resource (remote server, or o365 endpoint) is expensive. Lastly, who wants to actually sit there and manually run these commands when they could be automated and invoked on demand via other means… such as via a web-service etc. Hence this is how stateful-process-command-proxy came to be… it provides the building block bridge between node.js and a pool of pre-established Powershell consoles. Once you have node.js talking to stateful-process-command-proxy, you can build whatever you want on top of that in node.js to mediate the calls.
The first higher level NPM module that builds on stateful-process-command-proxy is powershell-command-executor
What this adds on top of stateful-process-command-proxy is probably best described by this diagram:
So the main thing to understand is that the module provides the PSCommandService class which takes a registry of pre-defined “named” commands and respective permissible arguments. The registry is nothing more than a object full of configuration and is easy to define. You can see an example here in the project which defines a bunch of named “commands” and their arguments usable for o365 to manipulate users, groups etc. PSCommandService is intended to serve as a decoupling point between the caller and the StatefulProcessCommandProxy… in other words a place where you can restrict and limit the types of commands, and arguments (sanitized) that can ever reach the Powershell processes that are pooled within StatefulProcessCommandProxy.
It is PSCommandService‘s responsibility to lookup the named command you want to execute, sanitize the arguments and generate a literal Powershell command string that is then sent to the StatefulProcessCommandProxy to be execute. StatefulProcessCommandProxy, once the command is received is responsible for checking that the command passes its command whitelist and blacklist before executing it. The sample o365Utils.js config file provides a set of pre-canned (usable) examples of init/destroy commands, auto-invalidation commands and whitelist/blacklist configs that you can use when constructing the StatefulProcessCommandProxy that the PSCommandService will use internally.
The next logic step is to expose some sort of access to invoking these pre-canned “commands” to callers. One way to do this is via exposing it via a web-service.
WARNING: doing such a thing, without much thought can expose you to serious security risks. You need to really think about how you will secure access to this layer, the types of commands you expose, your argument sanitiziation and filtering of permissible commands via whitelists and blacklists etc for injection protection. Another precaution you may want to take is running it only on Localhost for experimental purposes only. READ OWASPs article on command injection.
Ok with that obvious warning out of the way here is the next little example project which provides this kind of layer that builds on top of the latter: powershell-command-executor-ui
This project is a simple Node.js ExpressJS app that provides a simple set of REST services that allows the caller to:
- get all available named commands in the PSCommandService registry
- get an individual command configuration from the registry
- generate a command from a set of arguments
- execute the command via a set of arguments and get the result
- obtain the “status” of the underlying StatefulProcessCommandProxy and its history of commands
Given the above set of services, one can easily build a user-interface which dynamically lets the user invoke any command in the registry and see the results… and this is exactly what this project does via an AngularJS interface (albeit a bit crude…). See diagrams below.
Hopefully this will be useful to others out there, enjoy.
Hoping this will be useful for others out there, I’ve posted some code that could to serve as a lower level component/building block in a node.js application who has a need to mediate interaction with command line programs on the back-end. (i.e. bash shells, powershell etc.)
This is node.js module for executing os commands against a pool of stateful child processes such as bash or powershell via stdout and stderr streams. It is important to note, that despite the use-case described below for this project’s origination, this node module can be used for proxying long-lived bash process (or any shell really) in addition to powershell etc. It works and has been tested on both *nix, osx and windows hosts running the latest version of node.
This project originated out of the need to execute various Powershell commands (at fairly high volume and frequency) against services within Office365/Azure bridged via a custom node.js implemented REST API; this was due to the lack of certain features in the REST GraphAPI for Azure/o365, that are available only in Powershell (and can maintain persistent connections over remote sessions)
If you have done any work with Powershell and o365, then you know that there is considerable overhead in both establishing a remote session and importing and downloading various needed cmdlets. This is an expensive operation and there is a lot of value in being able to keep this remote session open for longer periods of time rather than repeating this entire process for every single command that needs to be executed and then tearing everything down.
Simply doing an child_process.exec per command to launch an external process, run the command, and then killing the process is not really an option under such scenarios, as it is expensive and very singular in nature; no state can be maintained if need be. We also tried using edge.js with powershell and this simply would not work with o365 exchange commands and heavy session cmdlet imports (the entire node.js process would crash). Using this module gives you full un-fettered access to the externally connected child_process, with no restrictions other than what uid/gid (permissions) the spawned process is running under (which you really have to consider from security standpoint!)
The diagram below should conceptually give you an idea of what this module does: process pooling, custom init/destroy commands, process auto-invalidation configuration and command history retention etc. See here for full details: https://github.com/bitsofinfo/stateful-process-command-proxy
Obviously this module can expose you to some insecure situations depending on how you use it… you are providing a gateway to an external process via Node on your host os! (likely a shell in most use-cases). Here are some tips; ultimately its your responsibility to secure your system.
- Ensure that the node process is running as a user with very limited rights
- Make use of the uid/gid configuration appropriately to further limit the processes
- Never expose calls to this module directly, instead you should write a wrapper layer around StatefulProcessCommandProxy that protects, analyzes and sanitizes external input that can materialize in a
- All commands you pass to
executeshould be sanitized to protect from injection attacks
- Make use of the whitelist and blacklist command features of this module
- WRAP this service via additional code that sanitizes all arguments to protect from command injection
Hopefully this will help others out there who have a similar need: https://github.com/bitsofinfo/stateful-process-command-proxy