Category: HTTP

Book review: Building Microservices

Screen Shot 2015-04-06 at 10.11.15 PMRecently I read Sam Newman’s “Building Microservices” , at ~280 pages its a fairly quick read. The reviews on this book overall are mixed and I can see where readers are coming from. By the title of this book one might expect some coverage of some of the microservices frameworks out there, concrete examples, maybe some actual code… but you won’t really find that here. Instead you will find a pretty good overview of various architectural approaches to modern application design in today’s world; covering general topics such a proper separation of concerns, unit-testing, continuous integration, automation, infrastructure management, service discovery, fault tolerance, high-availability and security etc.

In reality, none of the principles covered in this book are the exclusive domain of “microservice” application architectures, but rather can (and should be) applied to any application you are considering deploying; whether its a “monolithic” application or a suite of microservices interacting as parts of a larger functioning application.

In that right I think this book is definitely a good read and worth a look, if for nothing more than to ensure your team gets a refresher on good design principles and how they can be materialized with some of the newer frameworks and tool sets that have come out of our community in recent years. The material presented is sound.

Execute Powershell commands via Node.js, REST, AngularJS

Building on my last post on stateful-process-command-executor this post will cover how you can leverage that node.js module to expose the capabilities of Powershell cmdlets over a REST api presented through an AngularJS interface.  Why would one want to do this you ask? Well I’ve covered this in my last post but I will briefly explain it here.

(Note, what is described below could just as easily be built for Bash processes as well as the underlying module supports it)

The use case came out of the need to automate certain calls to manage various objects within Microsoft o365’s environment. Unfortunately Microsoft’s GraphAPI, does not expose all of the functionality that is available via its suite of various Powershell cmdlets for o365 services. Secondly when you need to do these operations via Powershell, its requires a per-established remote PSSession to o365…. and establishing (and tearing down) a new remote PSSession whenever you need to invoke a cmdlet against a remote resource (remote server, or o365 endpoint) is expensive. Lastly, who wants to actually sit there and manually run these commands when they could be automated and invoked on demand via other means… such as via a web-service etc. Hence this is how stateful-process-command-proxy came to be… it provides the building block bridge between node.js and a pool of pre-established Powershell consoles. Once you have node.js talking to stateful-process-command-proxy, you can build whatever you want on top of that in node.js to mediate the calls.

Layer one

The first higher level NPM module that builds on stateful-process-command-proxy is powershell-command-executor

What this adds on top of stateful-process-command-proxy is probably best described by this diagram:

 

So the main thing to understand is that the module provides the PSCommandService class which takes a registry of pre-defined “named” commands and respective permissible arguments. The registry is nothing more than a object full of configuration and is easy to define. You can see an example here in the project which defines a bunch of named “commands” and their arguments usable for o365 to manipulate users, groups etc.  PSCommandService is intended to serve as a decoupling point between the caller and the StatefulProcessCommandProxy… in other words a place where you can restrict and limit the types of commands, and arguments (sanitized) that can ever reach the Powershell processes that are pooled within StatefulProcessCommandProxy.

It is PSCommandService‘s responsibility to lookup the named command you want to execute, sanitize the arguments and generate a literal Powershell command string that is then sent to the StatefulProcessCommandProxy to be execute. StatefulProcessCommandProxy, once the command is received is responsible for checking that the command passes its command whitelist and blacklist before executing it. The sample o365Utils.js config file provides a set of pre-canned (usable) examples of init/destroy commands, auto-invalidation commands and whitelist/blacklist configs that you can use when constructing the StatefulProcessCommandProxy that the PSCommandService will use internally.

Layer two

The next logic step is to expose some sort of access to invoking these pre-canned “commands” to callers. One way to do this is via exposing it via a web-service.

WARNING: doing such a thing, without much thought can expose you to serious security risks. You need to really think about how you will secure access to this layer, the types of commands you expose, your argument sanitiziation and filtering of permissible commands via whitelists and blacklists etc for injection protection. Another precaution you may want to take is running it only on Localhost for experimental purposes only. READ OWASPs article on command injection.

Ok with that obvious warning out of the way here is the next little example project which provides this kind of layer that builds on top of the latter: powershell-command-executor-ui

This project is a simple Node.js ExpressJS app that provides a simple set of REST services that allows the caller to:

  • get all available named commands in the PSCommandService registry
  • get an individual command configuration from the registry
  • generate a command from a set of arguments
  • execute the command via a set of arguments and get the result
  • obtain the “status” of the underlying StatefulProcessCommandProxy and its history of commands

Given the above set of services, one can easily build a user-interface which dynamically lets the user invoke any command in the registry and see the results… and this is exactly what this project does via an AngularJS interface (albeit a bit crude…). See diagrams below.

Hopefully this will be useful to others out there, enjoy.

 

 

 

Executing stateful shell commands with Node.js – powershell, bash etc

Hoping this will be useful for others out there, I’ve posted some code that could to serve as a lower level component/building block in a node.js application who has a need to mediate interaction with command line programs on the back-end. (i.e. bash shells, powershell etc.)

The project is on github @ stateful-process-command-proxy and also available as an NPM module

This is node.js module for executing os commands against a pool of stateful child processes such as bash or powershell via stdout and stderr streams. It is important to note, that despite the use-case described below for this project’s origination, this node module can be used for proxying long-lived bash process (or any shell really) in addition to powershell etc. It works and has been tested on both *nix, osx and windows hosts running the latest version of node.

This project originated out of the need to execute various Powershell commands (at fairly high volume and frequency) against services within Office365/Azure bridged via a custom node.js implemented REST API; this was due to the lack of certain features in the REST GraphAPI for Azure/o365, that are available only in Powershell (and can maintain persistent connections over remote sessions)

If you have done any work with Powershell and o365, then you know that there is considerable overhead in both establishing a remote session and importing and downloading various needed cmdlets. This is an expensive operation and there is a lot of value in being able to keep this remote session open for longer periods of time rather than repeating this entire process for every single command that needs to be executed and then tearing everything down.

Simply doing an child_process.exec per command to launch an external process, run the command, and then killing the process is not really an option under such scenarios, as it is expensive and very singular in nature; no state can be maintained if need be. We also tried using edge.js with powershell and this simply would not work with o365 exchange commands and heavy session cmdlet imports (the entire node.js process would crash). Using this module gives you full un-fettered access to the externally connected child_process, with no restrictions other than what uid/gid (permissions) the spawned process is running under (which you really have to consider from security standpoint!)

The diagram below should conceptually give you an idea of what this module does: process pooling, custom init/destroy commands, process auto-invalidation configuration and command history retention etc. See here for full details: https://github.com/bitsofinfo/stateful-process-command-proxy

Obviously this module can expose you to some insecure situations depending on how you use it… you are providing a gateway to an external process via Node on your host os! (likely a shell in most use-cases). Here are some tips; ultimately its your responsibility to secure your system.

  • Ensure that the node process is running as a user with very limited rights
  • Make use of the uid/gid configuration appropriately to further limit the processes
  • Never expose calls to this module directly, instead you should write a wrapper layer around StatefulProcessCommandProxy that protects, analyzes and sanitizes external input that can materialize in a command statement.
  • All commands you pass to execute should be sanitized to protect from injection attacks
  • Make use of the whitelist and blacklist command features of this module
  • WRAP this service via additional code that sanitizes all arguments to protect from command injection

Hopefully this will help others out there who have a similar need: https://github.com/bitsofinfo/stateful-process-command-proxy

Processing ModSecurity audit logs with Fluentd

Recently had a need to take tons of raw ModSecurity audit logs and make use of them. First used Logstash and then attempted with Apache Flume (see previous articles). Next in line was Fluentd which is what this article is about, long story short I ended up just having to write a Fluentd output plugin to take the output from the tail multiline plugin and then format it into a more structured first class object that looks like the below example.

The Modsecurity Fluentd plugin is located here on Github: https://github.com/bitsofinfo/fluentd-modsecurity

  1. Get some audit logs generated from modsecurity and throw them into a directory
  2. Edit your fluentd config file and customize its input to use the tail multiline plugin and then the modsecurity plugin, an example is here.
  3. Customize your output(s)
  4. On the command line:  “fluentd ./fluent.conf -vv” 

This was tested against the latest version of Fluentd available at the time of this article.

The end result of it is that with this configuration, your raw Modsec audit log entries, will end up looking something like this JSON example below. Again this is just how I ended up structuring the fields via the filters, you can fork and modify the plugin as you see fit to output a different format, or even make it more configurable

EXAMPLE JSON OUTPUT, using https://github.com/bitsofinfo/fluentd-modsecurity

{
"modsec_timestamp":"08/Nov/2013:06:22:59 --0400",
"uniqueId":"C5g8kkk0002012221222",
"sourceIp":"192.168.1.22",
"sourcePort":"34156",
"destIp":"192.168.0.2",
"destPort":"80",
"httpMethod":"GET",
"requestedUri":"/myuri/x",
"incomingProtocol":"HTTP/1.1",
"myCookie":"myCookie=testValue",
"requestHeaders":{
"Host":"accts.x4.bitsofinfo2.com",
"Connection":"keep-alive",
"Accept":"*/*",
"User-Agent":"Mozilla/5.0 (Windows NT 6.1; WOW64) Safari/537.36",
"Referer":"https",
"Accept-Encoding":"gzip,deflate,sdch",
"Accept-Language":"en-US,en;q=0.8",
"Cookie":"myCookie=testValue; myapp_sec_7861ac9196050da; special=ddd",
"Incoming-Protocol":"HTTPS",
"X-Forwarded-For":"192.1.33.22"
},
"XForwardedFor":"192.1.33.22",
"XForwardedFor-GEOIP":{
      "country_code":"TW",
      "country_code3":"TWN",
      "country_name":"Taiwan",
      "region":"03",
      "region_name":"T'ai-pei",
      "city":"Taipei",
      "latitude":25.039199829101562,
      "longitude":121.5250015258789
   },
"serverProtocol":"HTTP/1.1",
"responseStatus":"200 OK",
"responseHeaders":{
"Vary":"Accept-Encoding",
"Expires":"Fri, 08 Aug 2014 10",
"Cache-Control":"public, max-age=31536000",
"Content-Encoding":"deflate",
"Content-Type":"application/x-javascript; charset=UTF-8",
"Set-Cookie":"zippy=65.sss31; path=/; domain=accts.x4.bitsofinfo2.com",
"Connection":"close",
"Transfer-Encoding":"chunked"
},
"auditLogTrailer":{
"Apache-Handler":"proxy-server",
"Stopwatch":"1375957379601874 39178 (989 4992 -)",
"Producer":"ModSecurity for Apache (http://www.modsecurity.org/); core ruleset",
"Server":"Apache (party6)",
"messages":[
{
"info":"Warning 1. Operator EQ matched 0 at GLOBAL.",
"file":"/etc/d4/modsechttp_policy.conf",
"line":"120",
"id":"960903",
"msg":"ModSecurity does not support content encodings",
"severity":"WARNING"
},
{
"info":"Warning 2. Operator EQ matched 0 at GLOBAL.",
"file":"/etc/d4/modsechttp_policy.conf",
"line":"120",
"id":"960903",
"msg":"ModSecurity does not support content encodings",
"severity":"WARNING"
}
]
},
"event_date_microseconds":1.375957379601874e+15,
"event_date_milliseconds":1375957379601.874,
"event_date_seconds":1375957379.601874,
"event_timestamp":"2013-08-08T10:22:59.601Z",
"secRuleIds":[
"960011",
"960904",
"960903"
],
"matchedRules":[
"SecRule \"REQUEST_METHOD\" \"@rx ^(?:GET|HEAD)$\" ",
"SecRule \"&REQUEST_HEADERS:Content-Type\" \"@eq 0\" \"phase:2,deny,status:406,t:lo",
"SecRule \"REQUEST_FILENAME|ARGS|ARGS_NAMES|REQUEST_HEADERS|XML:/*|!REQUEST_HEADERS:Ref",
"SecAction \"phase:2,status:406,t:lowercase,t:replaceNulls,t:compres",
"SecRule \"&GLOBAL:alerted_960903_compression\" \"@eq 0\" \"phase:2,log,deny,status:406,t:lower"
]
}

Dropwizard Java REST services

To sum it up; Dropwizard rocks.

I’ve done quite a bit of WS development both the on client side and server side; interacting with both SOAP, REST and variants of XML/JSON RPC hybrid services. For my latest project I need to expose a set of REST services to a myriad of clients: phones, fat JS clients etc. This application also needs to talk to other nodes or “agents” that are also doing work in a distributed cloud environment. The core engine of this application really has no need for a bloated higher level MVC/gui supporting stack and bringing that into this library would just be a pain. I’ve always like the simplicity of being able to skip the whole JEE/Spring/Tomcat based container stack and just do a plain old “java -jar”… run my application…. but the reality of being able to do that has been lacking… until now.

In looking at the available options for picking the framework to use (such as Restlet, Spring-MVC REST, Spring Data REST etc and others), I immediately became discouraged when looking at examples for setting them up; they are full of complexity and lots of configuration and generally require a full application server container to run within, which just adds further complexity to your setup.

Then I stumbled across Dropwizard by the folks at Yammer. I encourage everyone reading this to just try the simple Hello World example they have on their site. If you have any experience in this space and an appreciation and vision for decoupling; you will immediately recognize the beauty of this little framework and the power it can bring to the table from a deployment standpoint. Build your core app engine back-end library as you normally would, toss in Dropwizard, expose some REST services to extend your interfaces to the outside world; throw it up on a server and “java -jar myapp server myconfig.yml” and you are ready to rock. (they make this possible by in-lining Jetty). Create a few little JS/HTML files for a fat JS client, (i’d recommend Angular) and hook into your REST services and you will have an awesome little decoupled application.

Securing Foscam IP camera access over SSL with Apache reverse proxying

UPDATED: 5/7/14 Important security vulnerabilities for Foscam cameras

UPDATED: 9/27/13  (The solution below does not include audio support; for audio over stunnel please see this post over at warped.org)

Recently I was assisting a local business setup their Foscam IP cameras and make them remotely accessible for monitoring purposes from anywhere in the world. The particular models they had installed are the FI8910W line. These camera’s are pretty cool and for ~$100 retail they are a pretty good deal in my opinion. The cameras can be accessed via a browser over HTTP and also support a rudimentary HTTP/CGI API. However one of the biggest issues with these cameras security wise is the lack of SSL support. The embedded webserver on these things only supports HTTP and basic auth in the clear which, outside of your local network is not a good thing if your requirements is to be able to view/manage them remotely from over the internet.

One solution for this is to simply front all access to your cameras with a SSL secured reverse proxy. We did this using Apache’s mod_proxy. I’m not going to go into every detail of how to do this below, but the point is to give the reader a starting point. You can lookup the details on all these apache configuration specifics elsewhere on the web, there are tons of examples out there.

The example below would be for securing access to 2 (two) Foscam IP cameras on your local network, living on an example subnet 192.168.1.0. It assumes the local network is fronted by a router that supports port forwarding, which most consumer/business routers do. The end objective here is that when you access https://myproxy.host.com:10001 you will be accessing CAM1 and when you access https://myproxy.host.com:10002 you will be accessing CAM2.

Secondarily you can also set it up so that you could hit CAM1 at https://myproxy.host.com:10000/cam1/ and CAM2 at https://myproxy.host.com:10000/cam2/

  1. CAM1 IP = 192.168.1.100 listening on port 80
  2. CAM2 IP = 192.168.1.101 listening on port 80
  3. Reverse Proxy Server = 192.168.1.50 listening on ports 10000, 10001, 10002
  4. Router IP address: 192.168.1.1  configured with port forwarding as follows: Port 10000 -> 192.168.1.50:10000, 10001 -> 192.168.1.50:10001 and 10002 -> 192.168.1.50:10002

OVERVIEW

  • First off you need to setup a computer/server running Apache. The Apache webserver is available for almost every operating system known to man from linux to windows, to os-x. This server’s IP address is 192.168.1.50 and ensure that name based virtual host support is enabled as well as mod_ssl.
  • Next ensure that apache is listening on all the necessary ports (the 3 mentioned above). You will want to have Apache listen on a separate unique port for each IP Camera it is proxying access to, or at least one unique port if you are proxying the cameras of of a sub-path: For this example we are assigning port 10000 -> [CAM1 & CAM2 via sub-dir proxies], port 10001->CAM1 only and 10002->CAM2 only. Within your apache configuration you will want to ensure that you have statements like the following configured:
NameVirtualHost *:10000
NameVirtualHost *:10001
NameVirtualHost *:10002
Listen 10000
Listen 10001
Listen 10002
  • Now that Apache is configured to listen on the necessary ports, we need to configure the actual virtual hosts and the reverse proxying directives within each host, see the example below:
###############################
# Reverse proxy config for BOTH
# CAMs (1 & 2) via sub-paths
# @ 192.168.1.100
###############################
<VirtualHost 192.168.1.50:10000>
 ProxyRequests Off
 ProxyPreserveHost On
 ProxyVia On
 <Proxy *>
 Order deny,allow
 Allow from all
 </Proxy>

 # CAM1 (note trailing / is important)
 ProxyPass /cam1/ http://192.168.1.100:80/
 ProxyPassReverse /cam1/ http://192.168.1.100:80/

 # CAM2 (note trailing / is important)
 ProxyPass /cam2/ http://192.168.1.101:80/
 ProxyPassReverse /cam2/ http://192.168.1.101:80/

 CustomLog /path/to/apachelogs/access_cam1.log combined
 ErrorLog /path/to/apachelogs/error_cam1.log
 ServerName cam3

 SSLEngine On
 SSLCertificateFile /path/to/sslcert/mysslcert.crt
 SSLCertificateKeyFile /path/to/sslkey/sslkey.key

 <FilesMatch "\.(cgi|shtml|phtml|php)$">
 SSLOptions +StdEnvVars
 </FilesMatch>
 <Directory /usr/lib/cgi-bin>
 SSLOptions +StdEnvVars
 </Directory>

 BrowserMatch "MSIE [2-6]" \
 nokeepalive ssl-unclean-shutdown \
 downgrade-1.0 force-response-1.0
 # MSIE 7 and newer should be able to use keepalive
 BrowserMatch "MSIE [17-9]" ssl-unclean-shutdown
</VirtualHost>

###############################
# Reverse proxy config for CAM1
# @ 192.168.1.100
###############################
<VirtualHost 192.168.1.50:10001>
 ProxyRequests Off
 ProxyPreserveHost On
 ProxyVia On
 <Proxy *>
 Order deny,allow
 Allow from all
 </Proxy>
 ProxyPass / http://192.168.1.100:80/
 ProxyPassReverse / http://192.168.1.100:80/
 CustomLog /path/to/apachelogs/access_cam1.log combined
 ErrorLog /path/to/apachelogs/error_cam1.log
 ServerName cam3

 SSLEngine On
 SSLCertificateFile /path/to/sslcert/mysslcert.crt
 SSLCertificateKeyFile /path/to/sslkey/sslkey.key

 <FilesMatch "\.(cgi|shtml|phtml|php)$">
 SSLOptions +StdEnvVars
 </FilesMatch>
 <Directory /usr/lib/cgi-bin>
 SSLOptions +StdEnvVars
 </Directory>

 BrowserMatch "MSIE [2-6]" \
 nokeepalive ssl-unclean-shutdown \
 downgrade-1.0 force-response-1.0
 # MSIE 7 and newer should be able to use keepalive
 BrowserMatch "MSIE [17-9]" ssl-unclean-shutdown
</VirtualHost>

###############################
# Reverse proxy config for CAM2
# @ 192.168.1.101
###############################
<VirtualHost 192.168.1.50:10002>
 ProxyRequests Off
 ProxyPreserveHost On
 ProxyVia On
 <Proxy *>
 Order deny,allow
 Allow from all
 </Proxy>
 ProxyPass / http://192.168.1.101:80/
 ProxyPassReverse / http://192.168.1.101:80/
 CustomLog /path/to/apachelogs/access_cam2.log combined
 ErrorLog /path/to/apachelogs/error_cam2.log
 ServerName cam3

 SSLEngine On
 SSLCertificateFile /path/to/sslcert/mysslcert.crt
 SSLCertificateKeyFile /path/to/sslkey/sslkey.key

 <FilesMatch "\.(cgi|shtml|phtml|php)$">
 SSLOptions +StdEnvVars
 </FilesMatch>
 <Directory /usr/lib/cgi-bin>
 SSLOptions +StdEnvVars
 </Directory>

 BrowserMatch "MSIE [2-6]" \
 nokeepalive ssl-unclean-shutdown \
 downgrade-1.0 force-response-1.0
 # MSIE 7 and newer should be able to use keepalive
 BrowserMatch "MSIE [17-9]" ssl-unclean-shutdown
</VirtualHost>
  • Ok, so before you start up apache, you need to generate your own self-signed SSL certificate/key. See those lines above in the configuration for “SSLCertificateFile” and “SSLCertificateKeyFile”? You will need to generate your own SSL private key, certificate request, and then self sign it. The results of those openssl commands yield files that you point to on your local proxy server. You can read here for an example on how to generate the necessary files
  • Next ensure the router that sits in front of your proxy server @ 192.168.1.50 has port forwarding enabled and forwards traffic going to port 10000, 10001 and 10002 to your proxy server.
  • Start up apache, work out the kinks and you should be ready to go. If you are outside of your normal network you will need to find your router’s WAN public IP address and go to https://my.external.router.ip:10001 and https://my.external.router.ip:10002 and you will be accessing CAM1 and CAM2 respectively over SSL from anywhere in the world. OR secondarily you can also go to https://my.external.router.ip:10000/cam1/ and https://my.external.router.ip:10000/cam2/ to hit the cameras. Please note that traffic from your browser to your proxy server is encrypted with SSL, however the SSL encryption will terminate at the proxy server. Network traffic from your proxy server to CAM1 and CAM2 is unencrypted but only running over your local network. This article assumes you trust who is on your local network not to be sniffing packets.
  • You will also want to ensure that your proxy server has a firewall on it, permits IP forwarding, limits access to only the necessary ports and is configured securely. You can handle that on your own and that is outside of the scope of this article.
  • Hopefully this helps someone out there who wants to securely access their IP cameras over the internet. Note that what is described above should work with any IP camera on the market that only supports HTTP, however the general procedure described above was only tested to work with Foscam model FI8910W

SOFTWARE FOR VIEWING YOUR PROXIED CAMERAS

I’ve received many questions regarding which apps out there support talking to Foscam’s behind a SSL secured proxy and unfortunately the few I’ve used all fall short in one way or another. Proxying http/https based resources on a network (via ports, sub-paths or other methods) is a technology that has been around for a long time and from an client apps perspective it need not know it is even there. Secondarily, the Foscam camera APIs will work just fine regardless of how they are proxied (from the root url or off of a sub-path in the proxy). Regardless here are some apps I’ve used below with some notes

  • iOS: FoscamPro: Cool app, works great when you are on your internal network, but fails miserably if you try to use it from outside your network when your cameras are behind an SSL secured proxy as described above. Why? The FoscamPro application simply DOES NOT support SSL. (FoscamPro devs: PLEASE IMPLEMENT THIS!) The only way to use FoscamPro in the setup above is if you have a VPN server running behind your router; you then connect to your home VPN which lets you appear “internal” to your local network when you are outside of your network, and then access your cameras directly, bypassing the proxy. The VPN itself is what is encrypting all of your communications.
  • iOS: Live Cams Pro: Cool app, works very similar to FoscamPro but supports other manufacturers and more devices, generic url streams etc. They DO support SSL which works with the proxied setup described above. However they DO NOT support specifying a relative path off of the base IP that you are connecting to a Foscam camera with. This effectively eliminates your ability to proxy your cameras via sub-dirs (i.e. https://my.net/cam1/) which is CRITICAL if you have a lot of cams but your router limits the number of port forwards you can have! (Live Cams Pro devs: PLEASE IMPLEMENT THIS!)
  • Android: tinyCam Monitor PRO: Cool app, I must admit I am pretty sure this supports HTTPS as I was testing with this earlier this summer for the port-> cam based config. I have not tested with the sub-dir path setup. If someone can shoot me an update on this I’ll appreciate it. (I’ve switched to all iOS)

Easy Multipart file uploads / POSTs using URLStream or AS3HttpClientLib

When working on a cross runtime Zinc/AIR method method of posting files to a remote server, I came across Mike Stead’s URLFileVariable and URLRequestBuilder classes which makes the process of uploading multiple files (or ByteArray’s of data) via an HTTP POST much easier. Click here get his code

The basic concept is this: assuming you have one or more ByteArray’s of file data and the filenames to go with them, you prepare one or more URLFileVariable’s. The URLFIleVariable’s are added to the URLVariables as follows:

// create the standard "URLVariables"
var vars:URLVariables = new URLVariables(); 
// lets set a ByteArray of JPG data with the filename "myFile.jpg" against the POST variable "file1"
vars.file1 = new URLFileVariable(myByteArray, "myFIle.jpg");

Now at this point, as in my case, I am passing these URLVariables to either an AIR implementation of an HTTP client (using URLStream) or a Zinc implementation (using AS3HttpClientLib). This was all described here in another post.

In my AIR implementation of an HTTP client abstraction I use an URLStream and the code that deals with the URLVariables works something like the following.


/* Standard URLStream code that interacts with URLVariables
which contain Mike Stead's URLFIleVariables and uses his URLRequestBuilder */

var stream:URLStream = new URLStream();

// [ REGISTER for your stream listeners here...

/* use Mike Stead's URLRequestBuilder to properly build the 
URLRequest and encode it to contain all the multi-part data and filenames in the "vars" created above */
var req:URLRequest = new URLRequestBuilder(vars).build();

// ensure a POST because URLRequestBuilder only specifies POST if the variables contains files.
req.method = URLRequestMethod.POST; 

// define some URL to post to
req.url = "http://my.something.com.xyz/url_to_post_to.xyz";

// fire off the POST
stream.load(req);

// as the load completes, your event handlers for the stream should take over

Now on the other hand, if you happen to be using AS3HttpClientLib (as described in one of my other posts) you will be coding the equivalent of the URLStream code above, however using AS3HttpClientLib as follows. Here we do not use the URLRequestBuilder but simply manually retrieve the appropriate information from Mike’s URLFileVariable instance.

....

// our AS3HttpClientLib HttpClient
var client:HttpClient = new HttpClient();

...
// [Remember to setup your "client" event hooks as described in the AS3HttpClientLib docs here]

// create a new POST request
var postReq:HttpRequest = new Post();

// lets create an Array to hold all the different "parts" of this multipart post
var parts:Array = new Array();

/* lets push the URLFileVariable we created earlier into the parts Array 
according to how AS3HttpClientLib expects it, as a Part object. Here
we are simply creating a new Part and populating it with the data
that we defined up above when we created the URLFileVariable */

parts.push(new Part("file1",vars.file1.data, "application/octet-stream", [{name:"filename",value:vars.file1.name}]));

// create a multipart that points to the "parts" Array
var mp:Multipart = new Multipart(parts);
postReq.setMultipart(mp);
					
// create a URI
var uri:URI = new URI("http://my.something.com.xyz/url_to_post_to.xyz");	

// fire off the request
client.request(uri,postReq);

Thanks again to Mike Stead’s UrlFileVariable and URLRequestBuilder classes for helping me make an generic HTTP multipart post process easier and I hope my post will give those reading some help along the way on implementing their own.

Advanced HTTP operations in Flex outside of AIR

I am currently pretty deep into a Flex/AS3 RIA desktop app project whereby I have several advanced needs. Such as to download partial files to the desktop (byte-range requests), execute HEAD requests to get remote file sizes, execute multipart POSTs, talk to some REST apis, and finally be able to read any write HTTP headers….. outside of AIR? Good luck with URLStream, URLRequest, URLRequestMethod and URLRequestHeader

Due to security sandbox restrictions, unless your application is running within Adobe AIR you are restricted to simple GETs and POSTs. You are also NOT allowed to touch the following headers:

Accept-Charset, Accept-Encoding, Accept-Ranges, Age, Allow, Allowed, Authorization, Charge-To, Connect, Connection, Content-Length, Content-Location, Content-Range, Cookie, Date, Delete, ETag, Expect, Get, Head, Host, Keep-Alive, Last-Modified, Location, Max-Forwards, Options, Origin, Post, Proxy-Authenticate, Proxy-Authorization, Proxy-Connection, Public, Put, Range, Referer, Request-Range, Retry-After, Server, TE, Trace, Trailer, Transfer-Encoding, Upgrade, URI, User-Agent, Vary, Via, Warning, WWW-Authenticate, x-flash-version.

That’s basically all of the ones I needed to touch…..so what to do? In my case I was developing some file management functionality that runs in both Adobe AIR as well as MDM Zinc. Amongst other protocols such as FTP, I needed to be able to pull file updates over HTTP as well (+authentication, HEAD checks etc). So I had to create an generic IHttpClient interface abstraction which then allowed me to implement runtime specific clients. So for the AIR side of things I was good to go by using all of Adobe’s URL* classes right out of the box and they work great for the AirHttpClient implementation of my interface. However for my Zinc client I was still stuck, until I ran across a fantastic little HTTP library out there that provides a custom HTTP library built on top of the Flex socket stack.

The AS3 library that ended up being used in my ZincHttpClient was: as3httpclientlib. as3httpclientlib solved all of my problems when running outside of the AIR environment. With as3httpclientlib you can do all of the following and more:

* GET, HEAD, PUT, POST, DELETE
* multipart/form-data (PUT and POST)
* HTTPS support using AS3Crypto TLS
* Post with application/x-www-form-urlencoded
* Reading chunked (Transfer-Encoding)

If you have the need to access a bit more advanced HTTP functionality and need to do it outside of the AIR runtime. I HIGHLY recommend as3httpclientlib. I have put this library through the works doing byte range requests, HEAD requests, header manipulation and downloading and posting all sorts of file sizes, the library works great. My only note is that it is a tad slower than URLStream based HTTP downloads under Adobe AIR, regardless, this little HTTP library is worth it. Kudos to the AS3HttpClientLib team!