Firefox File Out (nsIOutputStream) details - ArrayBuffer, String, and Array

A few helpful things to remember:

Goal: Write out binary data stored in a ArrayBuffer (html5 typed array) to a file

What I learned:

If you open a file with FileUtils.openSafeFileOutputStream, then the only way to properly close it is with FileUtils.closeSafeFileOutputStream. This is something the docs forget to tell you but very important.

Typed Arrays have bounds checking on instantiation. If byteLength%BYTES_IN_TYPE!=0 then an exception is thrown.

String are written to memory sequentially. Array indexes are not. TypedArray index locations can't be accessed directly (reading them actually reads their wrapper object, or to be more precise, it seems if you run "var a = buffer[i], b = buffer[i+1];" you would find that &a is not related to &b).

Writing out in an OutputStream will convert to a string whatever object you pass into the buffer parameter!!! The result of such a call (toString) will be what is set as the location in memory of the buffer, assuming typeof buffer.toString() == "string". If you could spoof that, that would be wearrryy wuseful.

Ways to write to a file
1) NetUtil.asyncCopy
2) nsIFileOutputStream
3) nsIBinaryOutputStream <--- <3 some love


I tried and tried and tried some more, to find some Firefox service or component which could just write the binary out WITHOUT conversion to a string first. Couldn't find one. They ALL took in a nsIInputStream only. Which sucks because my data isn't an input stream, AND I can't just implement nsIInputStream in javascript due to "Native Only" method calls.


So, any way to have a ArrayBuffer backed InputStream?? That would be nice. But no soup for me, it seemed.


I was just trying to avoid implementing my own buffered writer.  Anyways, given that FileOutputStream only accepts string input, and NetUtil.asyncCopy only takes a legit input stream, I've no choice but to convert to a string manually **(or so I thought).

So, given the items I learned above, this is what I had to do:



But but, right after I finished with this, it occurred to me that for every InputStream, there must be an OutputStream. Since there was a BinaryInputStream, I did a search and indeed, I found BinaryOutputStream. I'd no particular reason to believe that it would behave differently than FileOutputStream, but just to make sure, I tested using a Typed Array as the output. Low and behold, it worked!! Seems BOS runs through the index values of an object. I would wager that it actually creates an Iterator and runs through that, but I didn't test that.

So once I knew about BOS, my code simplified to using just service calls, which is nice.

Firefox modules - Implementing a module, specifically, ContentPolicy

In Firefox, Modules are isolated global environments. Imagine the bastard child of a WebWorker and a CommonJS/AMD import/require statements (see: http://addyosmani.com/writing-modular-js/).
As stated, Module are global (singleton) for the Firefox application, and are started (run) the first time they are imported into another (running) JS scope. Modules can be registered with Firefox when they start up.
In this example, we'll implement a nsIContentPolicy module, which can be used for intercepting (and blocking) ALL requests made from the browser.

To get started:

1) Create a chrome/modules folder. Create the module file, $(module_name).jsm, where module_name can be what you like.
2) Add to the chrome.manifest, where app_name is the same name used in the manifest already:  "resource    $(app_name)    chrome/modules/"
3) Add this code to where ever you want the module to be accessible from, where scope_reference is where ever you want the exported module objects to be exposed on: "Components.utils.import("resource://$(app_name)/$(module_name).jsm", $(scope_reference));"
4) Inside $(module_name).jsm, you'll need this basic skeleton code. Since the example module will implement nsIContentPolicy, I'll use the name "ContentPolicy":

Note: You must customize the  "classDescription", "classID", "contractID", and QueryInterface list. But, the QueryInterface list must include "Components.interfaces.nsIFactory"
5) Actually add and implement all the methods which you declare that you will implement (in QueryInterface list of Interfaces).
6) EXPORTED_SYMBOLS string array will add all objects on the module's scope (window) into the $(scope_reference) passed into the import call.

Registering the Module with Firefox - If you want FF to USE YOUR MODULE, then you have to REGISTER IT

For registering with Firefox, and for ContentPolicy in particular, we need to add the Module as a listener for certain events. SEE THIRD SNIPPET

Custom for nsIContentPolicy 

Must add these methods to implement nsIContentPolicy: SEE SECOND SNIPPET

Annoying differences in how browsers handle Flash movement

Everyone hates the differences between the different major browsers out there. Not to mention the different versions of those browsers. Supporting all of them is a pain in the ass. And here is just one more reason:

Action: Take a Flash Object element on a page, and then move it someplace else (append the element to a different parent node)

Code:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17

var newCont = document.createElement("div"); var swfOb = document.getElementById("swf-object-element"); //already exists on the page print(swfOb.lastChild); //<param name="flashvars" value="name=value" /> swfOb.lastChild.value = "name=value2"; document.body.appendChild(newCont); newCont.appendChild(swfOb); swfOb = document.getElementById("swf-object-element"); var swfExternalFunction = e.setCorner; e.setCorner("tr"); //(attempts to) move the corner to the top right (starts in the bottom right) //In NO browser, does that call actually WORK //...wait 100ms swfExternalFunction == null //false IE,FF;  true Chrome swfExternalFunction == e.setCorner //false!! all e.setCorner("tr") //Error! IE;  works in FF,Chrome swfOb.lastChild //<param name="flashvars" value="name=value" />  FF!! ; <param name="flashvars" value="name=value2" />  Chrome, IE (sometimes)

Consequences:

SWF is in the bottom right. The call to "setCorner" does nothing. All browsers. 

Because::

IE: Will not let you call a Flash External Interface function on a moved element. BUT!!!! the function will still exist on the element. Buggars. And I couldn't get a straight answer from IE when modifying the Object Element before/after the location change. IE seems to have some sort of race condition going on. Sometimes it does, sometimes it doesn't, and sometimes the buggar doesn't even allow ANY External Interface any more. More hate for IE. Tested in IE 10, _but_ in IE8 standards mode. I had similar problems in actual IE8 - I could not get consistant results. I need to create better tests for IE, but that's time I don't have.

FF: FF re-inits the SWF, just like Chrome. Any changes to the "old" element will NOT be reflected in the new one. Whats WORSE is that the reference to the element's External Interface function STILL EXIST (on the Object) while the "new" object is being initialized, BUT!!! those functions STILL REFERENCE THE "OLD" OBJECT!! Talk about craziness... 

But wait, I'm not done. Firefox completely IGNORES changes to the object element in the mean time. So you can't, say, change the FlashVars and have them be updated in the "new" Object. Booooooo FF

Chrome: The best of the three. I'd still prefer that the SWF not be re-initialized, but oh well. At LEAST Chrome doesn't lie to you and expose functions that are out of date or don't exist. As well, it recognizes changes to the Object Element and on the "new" Object, those changes are present.

RESULT:

The only sure fire way to move a SWF element is to just set a timeout for after the move, and re-init it then, knowing that all your previous changes are gone. Which means you have to keep state externally, or get the current state from the Object before you move it. There's NO WAY to check to verify that it's "ok yet" to make changes after you've moved it, so you just have to set a long timeout and pray that it's long enough (iff you don't want to make browser specific code, that is).

Notes and Rants about Firefox and Chrome Addon building

Originally from April 16, 2012

When I first started building an Addon for Firefox, I thought it was really cool to have access to the inner workings of the browser via their extensive list of API's. I was not used to such openness by an application - usually even the most open application will limit developers in some very important and annoying ways. As a Flash/Actionscript developer, I can rant several days about how annoying a few but very important limitation in Flash are and how many weeks of my life I've devoted to getting around them. This was not the case with FF and I appreciated it. 

But after a few months, I started to get a real picture of how things really worked - every API was different, separate, and likely built by different people. This meant that every time I had to use a new API, there was a learning curve. Then months later debugging, I would have to go back and remember how the h3ll I did that. MDN (Mozilla development reference pages) became my new bible and indispensable too to understanding what would otherwise never makes sense to me. Now a few parts of the way FF extensions work was interesting to me - I really like how modules work and I appreciate the design, as well their DB API was fairly convenient and easy to use, but most curiously, is how they allow development in both C and JS and how the two relate to each other (FF was initially C only but they soon built a layer on top of that to allow for JS coding - early code examples show both a C and JS version, which allows you to glimpse at how they must have translated between the two) . 

Building this all in JS, back when the only alternative was IE and  C/C++, must have made (non-c) developers weep with joy. But unfortunately there are too many artifacts of that early transition that just aren't necessary in JS, but that they keep in place for backwards compatibility? easy understanding by long term FF programmers? simply "that's how we've always done it"?? At least these annoying quirks in FF are well documented, if still confusing and time consuming to implement. Of course that's not including the time it takes to understand each API and more importantly, what the data received from actually means / sent to it actually does ( I found an annoying lack of the details of this type of information on MDN - they explain well how to code for it, but they often neglect the details about the specifics of what the API actually does).

So after a year of my time (9hrs a day, 5 days a week: felt like a lifetime) devoted fully to Firefox development, 3 full revisions of the same plugin and 10000 changes, I, and my employer, now have a stable, well working Addon that I can be proud of. Another major reworking of it is in development, which I can say, in design, is much simpler and has -many- fewer points of possible failure (ie. more elegant). But in the middle of that transition, I was tasked with creating a brother for this Addon in Chrome and IE, though the IE brother is going to be fairly retarded. Thus the process of porting to Chrome begins. 

I knew a little about Addons in Chrome from a year ago when I made a simple history keeping Addon (which only kept the history for domains which I specified, and on those domains scanned each page for certain key words, logging instances when those words appeared). The structure of Chrome is completely different and far less powerful than Firefox - Firefox uses Gecko, which is a full blown application platform, whereas Chrome just provides a few API's. Initially, I was less than wowed. In fact, when visiting the Google campus, I stood outside the Chrome building, and in a voice as loud as I dared, told them to extend their API's because they "sucked" haha. And in scope, yes they do, compared with Firefox. But I came to learn that for what they lacked in scale, they made up for big time in at least having a unified design and a much simpler interface than FF. And fortunately, a HUGE feature of the Addon which I make, the ability to hear and intercept outgoing and incoming requests - was being implemented as I spoke (and boy, is it a million times simpler than the FF equivalent).

Chrome has inherit limitations on what features it can /should provide based upon how Addons are implemented. In Chrome, Addons are simply invisible html pages that are allowed to interact with other html pages the user has open. At the same time, they have a few special privileges in regards to loading information from the internet, as well, Chrome has a few convenient API's by which it can pass information back and forth from the Addon. First major design point - All communication in Chrome is done via JSON. Now, they are nice enough to do the translation for you, automatically calling JSON.parse and JSON.stringify when necessary, making it just a tad simpler for developers. This message passing has the same limitations as normal JSON services which populate the internet. But that also makes this very comfortable/familiar to web developers.  _  _ _ _  (( There are of course a few exceptions to this rule in recent API's, most notably the WebRequest API, which is the most recent API added to Chrome (and thank a deity too, because it's exactly what I needed for my addon). I don't mean that it's not JSON, I mean that it's not typical message passing. For those that don't know, "message passing" is an actual thing, not a noun+verb, and it is asynchronous, which as a feature has the single most significant impact on the design of an Addon for Chrome. And how the WebRequest API is different is that it acts synchronously to an addon's input. I doubt that Chrome developers suddenly decided to have a truly synchronous API, so I imagine it's simply designed to wait at certain times for the addon's response, but it still acts in sequence with external events, which makes it different from other APIs. )) _ _ _ _ _  So basically, everything is asynchronous, and everything is passed around as a JSON object. There are of course a few other quirks, but nothing else so important. Nothing is actually integrated with the browser or has control over any browser function/feature.

This is the exact opposite of Firefox. Firefox actually loads an Addon's code into the browser itself. Using "overlays", something like a CSS which "cascades" the different levels of code into a single combined source. This is then run as Firefox itself (actually, FF has two layers, the chrome and the gecko/xul runner backend ... addon code is added to the chrome). Once combined, an addon has full access to the chrome layer of Firefox and can pretty much do what it wants to that layer. As well, it's possible to create "modules" which are actually run as separate entities,  and even components which are attached to the Gecko backend (not integrated, just run on). The current reworking of the FF addon which I mentioned before, is actually transitioning from a mostly chrome layer based addon, to a purely module based form (aside from necessary UI). All functionality from the backend is done via API's, which as I mentioned before, are each different and occasionally very annoying to deal with. The lack of a single purpose, form, or style amongst these API's, means that not only are you stuck putting the pieces of the puzzle together, you also have to get your razor blade out and cut the pieces to fit each other. You would not -believe- the trouble involved in figuring out if a recently set cookie was a third party cookie or not!! You have to use three different API's and implement a component to do it correctly (certainly you can make some "guesses" and make this much simpler), and the component is terribly confusing because there are tons of different possibilities  no guarantees, and no documentation about the different cases you might see. Oh, don't forget the case where the user has multiple windows open!! Just shoot me now and get the suffering over with! Anyways, Firefox's addon capabilities are truly powerful, and truly scary at times. At least there are synchronous events (observations) and DB calls.

Then the time came to port the code of our mature FF plugin to Chrome. As I mentioned before, I had a negative opinion of Chrome's addon abilities due to it's lack of API's and limitations on functionality. But for my purposes, with Chrome's WebRequest API, it seemed to provide for the functionality I needed and thus would make a fully functioning Chrome addon possible. I'll not write my notes on this port here, but to make my point, I'll just mention that I found myself deleting significant portions of the FF addon source code. In some cases, a file of 300+ lines of code was reduced to 20 lines with Chrome with equivalent functionality. This was across the board - making the code easier to read and understand by those who are not me. Chrome actually seems to provide their API's with intended uses in mind, not just tossing some random information in the air and seeing what happens (FF). For example, just telling me what tab makes a http request is as simple as request.tab in Chrome - but in FF, I have over 80 lines of code to figure out the same thing (simple cases are about 15 lines, but it's the edge cases that really piss me off) and its STILL not guaranteed to find an answer. I'm just gonna give a "WTF?" on that one. 

Though my main concern of finding out what web page set a third party cookie, is about equally difficult in Chrome - as just like FF, the only information you start out with is a domain name of the cookie - the rest you have to figure out yourself. I'm gonna give a "WTF??" to both Chrome and FF on that one. I expect as much from FF, but I'm surprised that Google didn't think that's important information.

Details on Secure Loading of JS Modules and Event Handling in Various Browsers

PRIMARY GOAL:  Load in external scripts which cannot be read/accessed by other scripts in the SAME context, nor who's loading can be stopped. Special care must be taken that no script can fake the load, inject their own script, or hijack listeners.

Sub-Goal 1 - Add a script element to the DOM without ANYONE being able to detect it.

Sub-Goal 2 - Give the script element a SECRET key (that NO other party can read) WITHOUT using the server side

Secondary Goal  - Securely be able to identify the ACTUAL PARTY of the current script (ie not just the context it's from). As well, identify all parties in the call path of the currently running script in order to verify that no external party is making the current function call.

Actions I'm taking - Add a script element to the DOM (HEAD), with src = external script. For sub-goal 2, I'm appending a url parameter = key to the src url of the loaded script.

Detection Mechanisms Used to Defeat Security:

Poller  

setInterval for 1 milli and get all "script" elements on page, search for the new element.

Listeners

ALL Dispatched Events in this process (and thus listened for, listeners attached to the HEAD)

• readyStateChange
• load
• DOMSubtreeModified
• DOMNodeRemoved
• DOMNodeInserted
• ?? DOMNodeInsertedIntoDocument
• ?? DOMNodeRemovedFromDocument   

?? are these supposed to be dispatched?? in no instance did I detect them. I attached these particular listeners to the document as well.

Browser Support

Minimum Targeted Browsers:

FF 3.6+, IE7+, Safari 5.2+, Chrome 12+

Actual Tests Run on:

FF 3.6, FF 4, FF 6, FF 17, Safari 6.0, Chrome 18, IE7, IE8, IE9, IE10

Non-Targeted or Tested, But Likely to Work on due to what I read:

FF2.0+, IE5+, all Chrome

Critical Feature Support with Tested Browsers 

DOM Mutation Events : all except IE8-

ReadyStateChange attribute on script element: IE5+

addEventListener : all except IE8-

document.currentScript : FF4+

Error.stack : all except IE9-

Removal of SCRIPT Element from DOM does NOT affect the running/event dispatching of the script : all except IE8-

Results : Secondary Goal

This goal is made possible through the call stack which exists on Error objects. With the Error.stack property, ALL scripts in the call stack are listed, including their SOURCE, function name, and line number. If this Error.stack string is parsed, then the actual URL of the parties in the stack can be accurately determined. The current party would be the top entry in this stack. The party which initiated the call would be the bottom most party. Thus satisfying Goal 3.

Restricted to browser with Error.stack. 

Results : Sub-Goal 2

Add a secret key to the URL of the loaded script. Add the element to the DOM. Immediately change the src to something else. The loaded script can later pick up the original URL from the stack trace. This isn't really intended to be used in conjunction with the other methods on this page, as it would be a bit "overkill". You could though, if you were paranoid.

With the Error.stack property, all scripts in the call stack are listed, including their SOURCE, function name, and line number. Critical in this is that the source mentioned in the stack is unrelated to the current "src" attribute of the SCRIPT element to which the script belongs, meaning that if the "src" property is changed on the Element, then the call stack itself is the ONLY way to access the original source URL from which the script was loaded. If the loaded script is protected - meaning it has no globally accessible functions and no callbacks which can be accessed by a page script - then this URL becomes private between: the script which set the source, the external script itself, and any script which the external script calls. Thus satisfying Goal 2.

Also key to this is that no other party be able to access the Element's "src" before it can be changed (hidden) and that even after changed, the original "src" is the only URL actually used. This is arbitrary in non-IE, but in IE, you have to let the call stack complete before changing anything on the Element. My research confirmed that the "DOMNodeInserted" event on the new SCRIPT element is the first event thrown when adding the element to the DOM, and thus is the first place possible to change the "src" property. It appears that doing so here does not affect the URL which is loaded (the new URL is ignored by the browser) so no additional work is needed. For browsers without DOM Mutation Events, the Poller is the only way** to detect new Elements added to the DOM, therefore as long as the SCRIPT element is removed in the same call stack as it is added, then the Poller cannot detect the addition.

Goal success restricted to browser with Error.stack.

** as well as onpropertychange, but since this does not get dispatched till the element is removed and contains no reference to the new element, is not further considered.

Results : Sub-Goal 1

Sub-Goal 2 is lamost pointless if you can achieve Sub-Goal 1, since the existence of the new external script is completely hidden and it can be guaranteed that "load" event listeners on the element will run immediately after the external script. As such you could set a predetermined (preferably self-destructing) variable or function for the listener to access. As it is, Sub-Goal 1 cannot be completely achieved thanks to Firefox 5+ behaving differently than the other browsers, and the next paragraph is understood to exclude FF5+.

Basically, there are two ways to detect elements in the DOM - event listener and just scanning the DOM (Poller). Adding event listeners on the new SCRIPT element and stopping the event propagation is sufficient enough to block all DOMMutation events from reaching it's parent node on the DOM. Even DOMSubtreeModified does not get dispatched on the HEAD element since the added SCRIPT element is removed in the bubbling state of it's DOMNodeInserted event - likely there is some mechanism which checks for a difference in the DOM subtree, and when it finds none, does not dispatch a SubtreeModified event***. So with all events being stopped, the only mechanism remaining is the Poller, which as mentioned before cannot detect elements if they are added and removed in the same call stack - and fortunately adding and removing in this manner does not effect the load or function of the external script, except in IE8-, and in that case, repetitive adding-removing till the external script is eventually loaded works.

Firefox 5+, however, works differently. When stopping the propagation of the DOMNodeInserted/DOMNodeRemoved events, the DOMSubtreeModified is guaranteed to be dispatched on the parent (HEAD) element, likely due to a lack of comparison checking (as mentioned at ***). Since the event cannot be stopped, nor can it be guaranteed that it be canceled before it reaches another listener (FF12< don't have stopImmediatePropigation so it's actually guaranteed to reach all listeners), this is a failure. Listeners then have access to the SCRIPT element and can add listeners or change the "onload" property of the element (which would guarantee to run before other listeners). Therefore in this case, the additional security measure of Sub Goal 2 is needed to provide authenticity to the function caller.

Success restricted to browsers not FF5+

Primary Goal

The main goal would be easy if Sub - Goal 1 was complete. But it's not. It's a simple thing to create a private scope in both the loader and loaded script, the only complexity is how to provide for secure communication between two completely private scopes. This can be done thanks to event listeners. The loader, adding a listener for the loaded script, as long as it can maintain its position as only (or at least first) event listener, is guaranteed to run immediately after a loaded script runs. Therefore if the loaded script provides a one-time-only accessible function, then there can be secure interaction. All this hinges on the loader's callback being guaranteed to run first. For all browser which can provide for Sub Goal 1, this is no problem since another script is never given an opportunity to attach a listener. However, for FF, we have to find another way.

The problem isn't really with getting access to the SCRIPT Element, the problem is guaranteeing that the loader's listener gets called first. Now most browsers call back listeners in the order in which they register - but this is not part of the HTML spec, so cannot be relied on. The only thing we can rely on is that the "onload" property, if set to a function, will be called first. Therefore we have to guarantee that the loader's callback is that function (or at the very least that this property is empty so that no one can sneak in front of the loader's callback). I have seen, in FF at least, the peculiar behavior that only if a "onload" is set when the element is added to the DOM, will the "onload" be called after the loaded script runs. Did you get that? Therefore simply by not setting the "onload" property we deny that ability to would-be hijackers. If you'll notice, all security feature have so far been on the loader's side, but if one was inclined to, needlessly and just for piece of mind, one could set the onload property of the document.currentScript to null from the inside the loaded JS.

This does not prevent other scripts from adding an Event listener and hoping that it is called first, but, after rummaging around the Firefox source code, I confirmed that listeners are stored in a simple array and would always be run "in order".  (Of course FF could change their source at any time, and in this, what I would consider so unlikely as to not need to be considered, event, we could use the security key of Sub Goal 2, for the loaded script to deny all access to those without that key. True, the hijacking script, if all these unlikely things happen, could then remove that exposing function, blocking all access to the loaded script, at least it can't do anything worse. Even if it removed the access function and replaced it with its own, thanks to Goal 2, to create some sort of callback which forces the call stack back into the loader script where it can verify the identity of the function which it is calling, is trivial to do - again, worse case scenario, only blocking the consumption of the loaded script.) So as things currently stand, even though the element get's exposed in FF5+, and FF10- can't stop event propagation immediately, still ok, because listeners get run in order and given no "onload" property, the first will always be the intended script.

So basically even though we can make it more secure using a private password and stack detection, we don't need to. However, stack detection will be key in security post-load.

Notable Notes

FF 3.5 - 3.6.x DONT load inserted script elements asynchronously. They make the whole page wait. Fix = add "defer" attribute to the element.

FF <3.5 still dont load an inserted script asynchronously, but there's no fix for that.

IE<=8 you can't "delete" references on the "window" object, 'cuz that throws an exception. And declared variables on that window object are not enumerable, which makes that first part "ok" as far as this project is concerned.

Tricks

add and remove it, over and over, never letting the element stay in the DOM outside the call stack - this eventually loads the script in IE 8-

1 2 3 4 5 6 7 8 9 10 11 12 13

document.head.appendChild(loadedScript.element); if(loadedScript.element.parentNode){   document.head.removeChild(loadedScript.element);   (function M()   {      if(!loadedScript.isLoaded){         document.head.appendChild(loadedScript.element);         loadedScript.element = URL+"";         document.head.removeChild(loadedScript.element);         setTimeout(M, 10);      }   })(); }

oddly, the onreadystatechange event is dispatched first before the loaded script has run!! OMg! So the listeners must be made to detect whether the loaded script has run or not.

get the URL of the current script (or any script in the call stack) when Error.stack exists

function(){   var S;   try{_}catch(e){     if(S=e.stack)       return doSomething(/:([^:]+\.[^:]+):/.exec(S.substr(S.lastIndexOf(S.indexOf('@')>=0?'@':' at ')))[1]);   } }

IE9- sucks because can't do this. Awesome, right?