Tagged: Javascript

Reacting to filesystem io events with Node.js

I recently was working on a larger ETL process that started with the reception of various data files via SFTP that were delivered on varying schedules. The requirement was that as files are received we generate a unique event in a database, then execute a sequence of commands to archive the files out of the delivery directory and offline to a central immutable annotated file repository.

This new functionality had to integrate with an existing SFTP legacy server, and would likely have other uses outside of this initial use-case.

Looking around for simple solutions based on a scripting language, I really could not find any that would work or be extensible enough for the need. Hence I ended up writing io-event-reactor.

The basic concept is this; you have a monitor that listens for IO events for particular paths on the filesystem. As these IO events occur, they are passed on to one or more evaluators to decide whether or not the IoEvent should be reacted to by one or more configured reactors. The entire latter sequence is encapsulated in an IoReactor instance that manages the flow between the three described components.

diag1

With this module, you construct and configure a single IoReactorService which can manage and contain one or more IoReactor instances, as many as you wish, providing for lots of flexibility for reacting to filesystem events.

When you configure the IoReactorService and its IoReactor instances, you specify which plugins you would like to use to fulfill the monitor and reactor roles. For evaluators you simply provide one or more functions which evaluate whether or not an IoEvent should be passed on to one or more reactors.

https://github.com/bitsofinfo/io-event-reactor

The default monitoring plugin is implemented using the great Chokidar library at: https://github.com/paulmillr/chokidar

For reactor plugins, I developed two based on my initial needs.

For an real-world example of the kind of application you could build on top of this, check out io-overwatch (albiet a simple utility) at: https://github.com/bitsofinfo/io-overwatch

Advertisements

Execute Powershell commands via Node.js, REST, AngularJS

Building on my last post on stateful-process-command-executor this post will cover how you can leverage that node.js module to expose the capabilities of Powershell cmdlets over a REST api presented through an AngularJS interface.  Why would one want to do this you ask? Well I’ve covered this in my last post but I will briefly explain it here.

(Note, what is described below could just as easily be built for Bash processes as well as the underlying module supports it)

The use case came out of the need to automate certain calls to manage various objects within Microsoft o365’s environment. Unfortunately Microsoft’s GraphAPI, does not expose all of the functionality that is available via its suite of various Powershell cmdlets for o365 services. Secondly when you need to do these operations via Powershell, its requires a per-established remote PSSession to o365…. and establishing (and tearing down) a new remote PSSession whenever you need to invoke a cmdlet against a remote resource (remote server, or o365 endpoint) is expensive. Lastly, who wants to actually sit there and manually run these commands when they could be automated and invoked on demand via other means… such as via a web-service etc. Hence this is how stateful-process-command-proxy came to be… it provides the building block bridge between node.js and a pool of pre-established Powershell consoles. Once you have node.js talking to stateful-process-command-proxy, you can build whatever you want on top of that in node.js to mediate the calls.

Layer one

The first higher level NPM module that builds on stateful-process-command-proxy is powershell-command-executor

What this adds on top of stateful-process-command-proxy is probably best described by this diagram:

 

So the main thing to understand is that the module provides the PSCommandService class which takes a registry of pre-defined “named” commands and respective permissible arguments. The registry is nothing more than a object full of configuration and is easy to define. You can see an example here in the project which defines a bunch of named “commands” and their arguments usable for o365 to manipulate users, groups etc.  PSCommandService is intended to serve as a decoupling point between the caller and the StatefulProcessCommandProxy… in other words a place where you can restrict and limit the types of commands, and arguments (sanitized) that can ever reach the Powershell processes that are pooled within StatefulProcessCommandProxy.

It is PSCommandService‘s responsibility to lookup the named command you want to execute, sanitize the arguments and generate a literal Powershell command string that is then sent to the StatefulProcessCommandProxy to be execute. StatefulProcessCommandProxy, once the command is received is responsible for checking that the command passes its command whitelist and blacklist before executing it. The sample o365Utils.js config file provides a set of pre-canned (usable) examples of init/destroy commands, auto-invalidation commands and whitelist/blacklist configs that you can use when constructing the StatefulProcessCommandProxy that the PSCommandService will use internally.

Layer two

The next logic step is to expose some sort of access to invoking these pre-canned “commands” to callers. One way to do this is via exposing it via a web-service.

WARNING: doing such a thing, without much thought can expose you to serious security risks. You need to really think about how you will secure access to this layer, the types of commands you expose, your argument sanitiziation and filtering of permissible commands via whitelists and blacklists etc for injection protection. Another precaution you may want to take is running it only on Localhost for experimental purposes only. READ OWASPs article on command injection.

Ok with that obvious warning out of the way here is the next little example project which provides this kind of layer that builds on top of the latter: powershell-command-executor-ui

This project is a simple Node.js ExpressJS app that provides a simple set of REST services that allows the caller to:

  • get all available named commands in the PSCommandService registry
  • get an individual command configuration from the registry
  • generate a command from a set of arguments
  • execute the command via a set of arguments and get the result
  • obtain the “status” of the underlying StatefulProcessCommandProxy and its history of commands

Given the above set of services, one can easily build a user-interface which dynamically lets the user invoke any command in the registry and see the results… and this is exactly what this project does via an AngularJS interface (albeit a bit crude…). See diagrams below.

Hopefully this will be useful to others out there, enjoy.

 

 

 

Executing stateful shell commands with Node.js – powershell, bash etc

Hoping this will be useful for others out there, I’ve posted some code that could to serve as a lower level component/building block in a node.js application who has a need to mediate interaction with command line programs on the back-end. (i.e. bash shells, powershell etc.)

The project is on github @ stateful-process-command-proxy and also available as an NPM module

This is node.js module for executing os commands against a pool of stateful child processes such as bash or powershell via stdout and stderr streams. It is important to note, that despite the use-case described below for this project’s origination, this node module can be used for proxying long-lived bash process (or any shell really) in addition to powershell etc. It works and has been tested on both *nix, osx and windows hosts running the latest version of node.

This project originated out of the need to execute various Powershell commands (at fairly high volume and frequency) against services within Office365/Azure bridged via a custom node.js implemented REST API; this was due to the lack of certain features in the REST GraphAPI for Azure/o365, that are available only in Powershell (and can maintain persistent connections over remote sessions)

If you have done any work with Powershell and o365, then you know that there is considerable overhead in both establishing a remote session and importing and downloading various needed cmdlets. This is an expensive operation and there is a lot of value in being able to keep this remote session open for longer periods of time rather than repeating this entire process for every single command that needs to be executed and then tearing everything down.

Simply doing an child_process.exec per command to launch an external process, run the command, and then killing the process is not really an option under such scenarios, as it is expensive and very singular in nature; no state can be maintained if need be. We also tried using edge.js with powershell and this simply would not work with o365 exchange commands and heavy session cmdlet imports (the entire node.js process would crash). Using this module gives you full un-fettered access to the externally connected child_process, with no restrictions other than what uid/gid (permissions) the spawned process is running under (which you really have to consider from security standpoint!)

The diagram below should conceptually give you an idea of what this module does: process pooling, custom init/destroy commands, process auto-invalidation configuration and command history retention etc. See here for full details: https://github.com/bitsofinfo/stateful-process-command-proxy

Obviously this module can expose you to some insecure situations depending on how you use it… you are providing a gateway to an external process via Node on your host os! (likely a shell in most use-cases). Here are some tips; ultimately its your responsibility to secure your system.

  • Ensure that the node process is running as a user with very limited rights
  • Make use of the uid/gid configuration appropriately to further limit the processes
  • Never expose calls to this module directly, instead you should write a wrapper layer around StatefulProcessCommandProxy that protects, analyzes and sanitizes external input that can materialize in a command statement.
  • All commands you pass to execute should be sanitized to protect from injection attacks
  • Make use of the whitelist and blacklist command features of this module
  • WRAP this service via additional code that sanitizes all arguments to protect from command injection

Hopefully this will help others out there who have a similar need: https://github.com/bitsofinfo/stateful-process-command-proxy

Scrollbar not appearing in IE7 when CSS overflow auto is set

Recently when working on a JQuery based UI, I encountered an annoying IE 7 problem. I had a scrollable DIV setup with overflow:auto; configured in CSS. The scrollable DIV had a one child div which contained text. As the text grows in length the parent scrollable div should trigger a vertical scrollbar. This worked fine in every browser, FF, SF, IE6/8 but would not work in IE7. When you first viewed the page in IE7 the text would just cut off and the scrollbar would not be visible, however when you clicked on another page element and brought focus back to the scrollable div, the scrollbar would then appear as expected. Annoying indeed. To fix this in IE7 I simply did the following in JQuery to dynamically adjust the height when the after the DIV gained focus and it fixed the problem. Might it help you? Who knows but it worked for me.

var h = $("myScrollingDivWithOverflowAUTOset").height();
$("myScrollingDivWithOverflowAUTOset").height(h+1);
$("myScrollingDivWithOverflowAUTOset").height(h-1);

Scathing report on the state of Dojo

Just came across this post today at Jeviathon regarding “What is the future of Dojo?”. Quite an interesting read and and I can understand the authors frustrations with the forums and lack of help on there, I experienced something similar last year when I posted several questions, only to never get an answer or I would search on what I was looking for only to find further lack of responses to the original posts. Keep in mind that during this time the big transition from <0.9 to 0.9 & 1.0+.

I have two general comments about Dojo.

A) My last experience with the toolkit was about a year ago as I was developing an AJAX based blogging plugin for use with the Grails framework . They key feature was that authenticated users, with permission to edit a particular post would simply be able to double click on the text to bring up a nice Dijit Dialog interface that presented the user with an inline Dijit Editor to make their changes. They could preview and commit their changes, all of that via AJAX so they did not have to leave the blog post itself. There was no need in this plugin to have a separate “admin interface” which is typical of many applications. This general behavior applied to editing/posting comments as well as adding new posts.

Overall I was pretty happy with Dojo and found existing examples on the web to be quite helpful. One great site that I frequented quite often was the Dojo Feature Explorer at DojoCampus. This site proved to be a much better resource then anything I could find on the forums.

dojobookB) During the time I was working on the Grails blogging plugin using Dojo, I was reading “Dojo: The Definitive Guide” which was an excellent book and I highly recommend it. (note this book may already be outdated with the 1.3.x Dojo releases)

Since that time I have used JQuery on a few projects. I am tending to lean towards JQuery since my experience with Dojo mainly because this seems to be the growing toolkit of choice for many other projects as well. (Well at least with Drupal, which is a CMS I have been evaluating recently) Secondly I like the way the code flows much better and it is quite easy to understand. Their documentation is very straightforward as well as their plugin repository which follows a consistent format.

Review: High Performance Web Sites

highperfwebThis is a review of “High Performance Web Sites” by Steve Souders

First off this book is written by someone who knows just a tad about front end client web interfaces and their performance, being the performance chief at Yahoo he has quite a bit of experience under his belt. Myself having done a fair amount of Javascript learned quite a bit by reading this book and I highly recommend it to anyone who is responsible for creating fast loading client web interfaces built with HTML and Javascript.

Some of the tips will be very familiar to folks out there such as minifying Javascript libraries, using a CDN and tinkering with your server’s expires headers, however some of the other tips are a bit less well known, such as putting JS library calls at the bottom of your page… why? Because all other resources, except JS files are downloaded in parallel, however a JS download by itself and nothing else will continue downloading until it is finished. The author does a great job of explaining several very helpful “under the hood” performance issues with browsers that you may have otherwise not known about. Get this book as it is a short read with a big bang for the buck.

Recommended? YES
Skill Level – Beginner to Advanced, however the advanced folks will get more out of the low-level HTTP and browser performance issues than beginners.

Review: Pro Ajax and Java Frameworks

proajaxThis is a review of “Pro Ajax and Java Frameworks” by Nate Schutta and Ryan Asleson

This book was written in 2006 and is targeting client side developers who want to add Ajax to their Java backed applications. The book tends to cater more towards an audience who likely wants an overview of various Ajax frameworks out there in order to get their feet wet or make a decision on which one to go with. The book does a good job by starting out with coverage of thee basic principles of Ajax and debugging tools etc. It then goes into specific Ajax frameworks, specifically Prototype, Scriptaculous, Dojo and Taconite. Sorry no JQuery here, when this book was written it was likely too new. The book then goes on to cover DWR with its own dedicated chapter. Towards the middle of the book the authors take a turn and focus more on Ajax integration with two Java front end frameworks, specifically Struts 1 and Tapestry. Finally the authors wrap up with Spring and JSF coverage. The latter two topics seemed a bit out of place.

Recommended? Skip it, the book is outdated by now. If you are unfamiliar with Java/Ajax integration and can get this book for under $10, you might consider it.

Skill levels: Beginner to Intermediate. Folks evaluating various frameworks and want an Ajax overview.