Phonegap / Cordova + AppCache

Cordova is great, right? Develop once, run everywhere etc etc. But what about updates? What about being backward compatible with all the new web services you are developing on the site, packaging the app and worrying about versions… no, we web developers can’t stand it.

First order of business – how to do updates. Common misconception is that traditional binary app stores (ios/android) don’t want the apps to self-update. It’s actually not true – you are allowed to self-update as long as the code you are updating runs in WebView. Bingo!

There are solutions on the market that give you such self-update mechanism, for example WorkLight that was recently acquired by IBM. It’s really not terribly hard to write this yourself if all you need is updates – get a zip file with all html/css/whatnot, download when application starts, extract, then actually render the UI.

However, there exists a system that is designed to do just that, and it’s arguably pretty good at that. The support for AppCache is very good both on modern desktop and mobile. The only question is how you’d use AppCache with Cordova.

Well, obviously your app can be a web site. Or web site can be the app… It’s really the same for web developers. I call most of the sites I create applications, because they are applications, from the customer point of view. Plus they can be pinned / clipped / made into an icon

I Steps on your site
1) Make sure you have offline manifest on the page that will be opened by Cordova
2) Double check – is there a redirect from the URL you think you are opening and actual URL that opens up? Appcache doesn’t work with redirects, so you have to have correct URL. For example, i had http://localhost/offline redirect to http://localhost/offline/ – and it didn’t work.
IIa Cordova iOS steps:
2) Allow Cordova to go out on the internet. In your project, go to Resources/Cordova.plist . Add record to External Hosts with your server name. You can just put star (*), but if someone makes your side redirect elsewhere you are in trouble – so only use for development.
3) Make sure cordova opens links inside its own webview, instead of safari, by changing OpenAllWhitelistURLsInWebView property to Yes.
IIb Cordova Android steps:
2) Allow Cordova to go out on the internet. In your project, open res/xml/config.xml and uncomment  <access origin=”.*”/> (this is only for development, see note in step 2 for iOS)
3) Add <preference name=”stay-in-webview” value=”true” /> to stay inside cordova
4) Adjust Android Cordova to enable AppCache. For some strange reason, it is not enabled by default for Android (while local storage and SQLLite are) – see setup() method in CordovaWebView.java file. It looks like in iOS UIWebView has it on by default and doesn’t let you mess with it. To fix this, just go into the only java file you should have in your app, under src/your.name.space/CordovaActivity.java (your file name could be different) and adjust the onCreate method to be like so:
public void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        super.init(); // Initialize appView, since it's only initialized inside loadUrl by default
        android.webkit.WebSettings settings = super.appView.getSettings(); // get WebView settings
        String appCachePath = this.getCacheDir().getAbsolutePath(); // Set path to appcache
        settings.setAppCachePath(appCachePath);
        settings.setAllowFileAccess(true); // Let browser write files - doesn't work without this
        settings.setAppCacheEnabled(true); // Enable app cache
        super.loadUrl("file:///android_asset/www/index.html");
}
Update June 4 2013: Per comments, this is fixed in Phonegap 2.5, so code above in #4 is not needed anymore (i.e. leave it alone the way cordova generates it, don’t modify)
III Finally, the Cordova app
This one is very simple, as you might imagine. In www/index.html all you need to have is:
<!DOCTYPE html>
<script type="text/javascript" charset="utf-8" src="cordova-2.0.0.js"></script>
<script>
    function onBodyLoad()
    {        
        document.addEventListener("deviceready", function(){
            location.href = "http://192.168.2.108/offline/";
        }, false);
    }
    </script>
<body onload="onBodyLoad()"></body>
Replace the URL with your local testing site that you can start/stop to test how offline is working, or with any other URL.
That’s it, start the app!

Actually using Cordova APIs

One caveat is that at least on Apple AppStore the wrapped  site has to be more than site. It has to have some additional functionality that is not available via the web. And that’s where Cordova APIs come to the rescue, with access to all the interesting APIs a regular WebView/UIWebView doesn’t have access.

The first thing you have to do is detect that web site was opened from inside Cordova. It doesn’t look like there’s currently an API in cordova.js to tell us it loaded inside the Cordova web view. The only semi-working solution is to detect that onDeviceReady didn’t fire – but you have to wait to determine this fact, and that will delay your page loading in regular browser. Easier way is to pass some querystring parameter, url?fromcordova=true , and then include javascript and execute ondeviceready handler based on this parameter.

Second, cordova javascript actually has two versions – one for iOS and one for Android (and I am assuming others for other platforms). File name is same, content is different. The reason is that on iOS it’s using custom URL handler gap:// to communicate with Objective-C side. On Android, it’s overriding exec, prompt and alert methods in JavaScript and then on Java side in DroidGap class it’s intercepting these calls and actually using them to interact with JavaScript side (especially prompt, that can return data back to JavaScript). So the platform ID has to be another parameter in your URL, for example: url?fromcordova=true&platform=iOS

 Note: the above setup instructions are for Cordova 2.0.0 – in previous versions some things still had PhoneGap names.

My experience with ExtJS

So more than a year ago I wrote about my new project – starting a re-write of our main application using ExtJS, AJAX and web services.

The project actually progressed very well (albeit I was the only developer on it, but that’s even more fun!). I have a proof of concept application, window and class framework to build on, a set of business classes and complete database schema for a good Point-of-Sale application. I have proved that a RAD environment can be set up and extended using the technologies I picked.

The modules I have created are:

  • Customer and item entry and search (with cool use of LiveGrid extension to view entire inventory or customer base)
  • Order search, creation, modification and saving
  • Sidebar for most important features
  • Theme changing (hey, it has to look cool)
  • Login and security system (using asp.net membership and roles on the server)

So far that’s all, I have actually stopped about 14 months ago, and started working on brand new project (more on that later).

I still think ExtJS is a very good framework, especially if you want to design a true AJAX app (not just spice up your old one). With all the FUD that was created by license switch, it’s hard to see the forest through all the trees. Yes, license changed, yes it was done poorly. But face it – it doesn’t make ExtJS any less cool as a technology, and it is worth the money they are asking (in my opinion anyways).

Word 2007 as blogging software

Microsoft Live Writer, cool software. An oddity of sorts for Microsoft – something that works well in v1. And works with multiple blog providers. And looks nice while doing that (with theme download and a bunch of addons).

But I can’t use it. I have a 130dpi monitor (one of those 1920×1200 17″ laptops – calculate your dpi here http://members.ping.de/~sven/dpi.html ), and of course everything is tiny by default. Windows 7 and all apps I use do a very good job scaling their UI, finally. Well, maybe it was the same with Vista, but I didn’t have high dpi. With NoSquint for Firefox (IE8 zooms in by default, based on Windows scale settings – nice touch) and all fonts in Visual Studio adjusted I really can’t tell difference between normal 80-90dpi and my 133dpi – it just looks much better (more pixels + ClearType). In some other software I use I also had to adjust font sizes (Trillian Astra, Skype, Outlook). Can’t adjust anything in Writer – I tried everything, even found where it downloaded my blog’s theme and tried changing size there – doesn’t work, well no obvious place to change it anyways.

So today I finally decided to write something in my blog *crickets*, and trying Word 2007 now. Setup wasn’t without a problem, as Word doesn’t include the nice auto-configuration setup of Writer. For those interested, /yourmtfolder/mt-xmlrpc.cgi is the path to the MetaWebLogApi script of MovableType. Couldn’t make Atom work for some reason, although apparently there is mt-atom.cgi.

As a UI, Word is definitely a step back from Writer – no themes, no addons, categories/tags don’t download. But you know what, I don’t that much care – I have good size font to write in, and that’s all I need. Will see how it handles pictures and VS formatting next.

New ExtJS + ASP.NET AJAX project

Being an SaaS company from the start (web technologies were chosen to drive our main product), it only made sense for us to eventually completely embrace AJAX. In addition, the original application is basically ASP.NET 1.1, with an inconsistent graphical design. So sprinkling UpdatePanel on top would just add some dynamical elements to an outdated UI design, without any real noticeable changes to the user.

This, among other things, was the reason to go all the way and redo our application using a better UI library – naturally ExtJS was my first choice – I didn’t want to invest too much into proof-of-concept application, or, frankly, lock us into one of proprietary component libraries. Version 2.0 just came out, and their component model is quite mature to build an enterprise LOB (line of business) application on top of it.

At the same time I want to keep some standards in place, and chose to use ASP.NET AJAX for the server side, in the web services area. That way we are developing same web service for our AJAX JSON application and for our SOAP integration, using same code in ASP.NET web service. In fact, it works transparently most of the time, and same service code can be accessed from both types of clients. One other thing it adds is nice JavaScript proxy object support on the client side. In the end, while we are developing the application, we get automatic web services API already developed for our partners.

Thirdly, to cut down on development time, an OR/M is used. Currently it’s CodeSmith’s .netTiers template – which automatically generates DAL, BOL and even "service" layer. More importantly, it creates web service (asmx) file, which can be used as-is, or copied from.

The database backend is MS SQL 2005, as always. However, I made some use of xml column type, based on previous experience, to try and accommodate some clients unique requests, which don’t mandate a separate table schema change. The end result is that the product is generic enough to accommodate different scenarios, and fast enough with little overhead from all the XML parsing – it is only needed when a feature sees minor use in real life (and yes, I do know about xml indexing, but for inserting/updating some overhead is still incurred).

The resulting tiers look like this:

  • ExtJS UI
  • client-side code
  • ASP.NET AJAX JavaScript proxy classes
  • JSON data transfer
  • ASP.NET web service
  • CodeSmith .netTiers business and data objects
  • MS SQL database

 

I am very careful to re-evaluate each of the technologies used in this list – an experienced developer may question each and every one of the choices. There are very good (and possibly better) replacements for ExtJS, ASP.NET AJAX JSON, ASP.NET web services, or .netTiers. But by taking the possible changes into account, it’s not that hard to design a system where all these things can be swapped later on. The only real investment that is made, is made into UI code and database designs. So, for example, any code that we custom-develop on top of OR/M can we migrated to a different OR/M solution – and the objects/field names are still driven by initial database design.

But things have to progress, so while designing for change, we move forward as quick as we can! That’s one of XP mottos – programming only what is needed today, and implementing it as simply as possible. And not being afraid to change later on.

So with that, I’ll finish this introduction into our next generation product, and get back to work :) There are tons of things to learn in all that new tech, and I will try to write about it – most of the user-created code can be found in ExtJS forums, and really all it takes is integration.

Multilingual Apache FOP

So today we got one of our arabic clients request – customize the order template to include the arabic text in the captions and text areas.

First thing i did was to try and just give FOP russian text – that just gets output as # signs. To get this working you need to follow this good old tutorial – it mainly talks about doing it manually (what i was doing first). The more extended, and newer (for versions 0.93 and 94) article is naturally located on official site. You will especially need it when you want to do it from code, not command line.

The font that you can use in Windows which covers almost all languages (per unicode) is located at c:\windows\fonts\arialuni.ttf . It’s a huge (22Mb) font that generates a pretty big font metrics (about 700k for me). This file, however, is not standard in windows – it seems to be part of the Microsoft Office installation. None of our windows servers have it by default, for example. Or you can just use plain arial, which covers all european scripts.

Finally tried doing arabic script using this font technique. This doesn’t work, and it doesn’t look like it will work any time soon. The problem of course is BIDI (bi-directional text support) – FOP can use arabic characters, but can not reverse the direction (and one can not simply reverse character order before giving it to FOP – arabic non-capital letters have to be connected to each other in a very special way). There’s no bidi support in FOP (although XSL-FO does have it in the spec). The mailing archive hints to some people working on it successfully, however the major obstacle seems to be lack of proper BIDI support in Java 1.3 itself – only 1.4 added needed methods, but FOP wants to stay 1.3-compatible.

Update: well, I just got a notification of 0.95 beta, and it does drop the 1.3 requirement. Good news! Hopefully the custom work that is done by other developers will be included eventually.

We had to revert to a workaround – the areas where arabic is needed are going to be bitmaps that our customers will be able to upload the images for.

Apache FOP XSL-FO implementation and footers

“flows” in XSL-FO can have static areas surrounding them, which are
naturaly used to make headers and footers on pages. Let’s consider
sales invoice like this:

sample invoice

Here, of course, you can position header and footer absolutely using
<fo:block-container left=”xx” top=”xx” position=”absolute”>.
However, when your invoice grows and you run out of space on the page,
and you want to see something like this:

invoice how it should page correctly

In XSL-FO standard this is achived using special static sections of the page:

<fo:page-sequence-master master-name="all-pages">
  <fo:repeatable-page-master-alternatives>
    <fo:conditional-page-master-reference 
            page-position="any" master-reference="any-page-id"/>
    <fo:conditional-page-master-reference 
            page-position="last" master-reference="last-page-id"/>
  </fo:repeatable-page-master-alternatives>
</fo:page-sequence-master>

Here you use master-reference parameter values your page master definition , for example:

<fo:simple-page-master master-name="last-page-id">

Inside that master you can define different footer. A good example is given here http://www.xmlpdf.com/ibex-examples.html

Problem is, tracking which page is “last” is very tricky in
free-flow documents. Take HTML for example, the common question about
how to make something 100% high is hard to answer. The content can flow
differently, and, more importantly, the element that is trying to make
use of that position can affect its own position. For example, now in
XSL-FO, a footer on the last page can be so high that the engine will
have to create another page after the page that was considered “last”
just to fit it. In which case, page before last will suddenly fit all
content on the last page too, because it doesn’t have footer anymore.
So engine has to make a decision on how to fix that in one or another
position! What I am getting to is that Apache FOP doesn’t do that yet,
and in general many complex things having to do with flow. It’s working
just fine with everything else it seems. This is why current version is
in “maintanence” now and developers are working on redesigning for next
version (which doesn’t have name yet, but either 0.30 or 1.0 ;-)

So what do you do in the mid time? Well, you have 3 workarounds
(none of which is perfect). First one that is normally suggested is to
use fo:footnote element to emulate the footer on the last page. Don’t
forget that you have to have footnote reference text, otherwise it
won’t generate anything:

<fo:block id="end">
<fo:footnote><fo:inline color="white">.</fo:inline>
<fo:footnote-body>...

The problem with footnote is that you can’t use absolutely
positioned blocks in it, and it seems to only like tables for fine
formatting. So the example that i gave above on the picture might be
hard to implement.

The second workaround is to use some code inside your program
(before FOP call) to tell which items will be on last page, and put
them into a different page master. That may be hard for some
applications, again, like in example I gave, each line in the invoice
can have wrapping text and as a result starts taking more space than
originally estimated.

The last solution is not perfect either – don’t make special
footer.This is what I ended up doing (and it was fine in my case, since
every invoice page has same footer in my clients case). So in the end
it looks like this:

result of implementing paging on invoice in FOP

You may think of switching to commercial XSL-FO generator, like
RenderX or Antenna House, which seem to support flows better, and
overall are high quality commercial products.

http://www.renderx.net/Content/tools/xep.html

http://www.antennahouse.com/product.htm

For my own software I am thinking of supporting many different
renderers, based on clients ability to buy commercial renderer or to
live with Apache FOP limitations (which will be fixed some day).

string.Trim(char[])

Apparently you should be more careful with Trim. How precisely it works i don’t yet understand, but the following operation:

variable.Trim(”’ “.ToCharArray())

does not always remove space or/and single quote from the ends of
the string. I think it is related to the fact that in that case it
doesn’t use Char.IsWhiteSpace, like if you use Trim without parameters.

Complex Sum in XSLT

When generating reports from XML data using XSLT I have to perform complex mathematical operations. Or, rather, operations that generic XPath can’t do. To achive that you can use variables in XSLT and fill node lists with them. So we have:

<xsl:variable name="totalCommission">
 <xsl:for-eachselect="orderz/accounts/payment/orderzs[@o_status != -9 and @o_prodtype != 'J']">
  <accum><xsl:value-of select="(@o_extprice  - @o_discount -itemmaster/itempricelist/@storecost*@qty) * ../../../../@salescommission div 100"/></accum>
 </xsl:for-each>
</xsl:variable>

as a result we create a variable that generates its element using some algorithm. After that, we can run a sum operation against it:

$<xsl:value-ofselect="format-number(sum(msxsl:node-set($totalCommission)/accum),'#,##0.00')" />

Very important in this operation is support of node set generator msxsl:node-set(), which is supported in MSXML that i use in this case.
To use it the root XML node will look like this:

<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform"version="1.0" xmlns:msxsl="urn:schemas-microsoft-com:xslt">

XML Cache

Developers frequently want to get data structures out of their code and into XML. But that bears a performance hit, since every time you read the file you have to parse it. In case of client applications you can avoid it by placing XML objects in the memory. However, in case of web-apps, memory gets destroyed after the request. Besides, you can’t monitor changes in the files themselves. ASP.NET introduced a very useful object called Cache, which is designed to help with precisely that problem. It stores objects on the application level (similar to Application object) but also allows you to monitor changes in the data. The following code is implementation of XmlDocument object storage and file change monitoring:

public static XmlDocument GetDOM(string path, Page page)
{
  if (page.Cache["xmlcache:dom:"+path] != null)
    return (XmlDocument)page.Cache["xmlcache:dom:"+path];
  else
  {
    XmlDocument x = new XmlDocument();
    x.Load(page.MapPath(path));
    page.Cache.Insert("xmlcache:dom:"+path,
      x,new CacheDependency(page.MapPath(path)));
    return x;
  }
}

The methods to store XslTransform and XPathDocument would look exactly the same. However, you should carefully monitor the contents of XslTransform document – CacheDependency object doesn’t monitor internal links of XSLT language, like xsl:import. By the way, I couldn’t find a way to automatically find all those tags in XSLT document and generate CacheDependancy for each of them.

Oh, and one other important thing. ASP.NET 2.0 extended on CacheDependency and introduced an object that monitors changes in SQL database! It’s ideal for many applications where you fill drop-downs with data from database, for example.

Simple Serializer/Deserializer

public static string SerializeObject(object obj)
{
 MemoryStream temp = new MemoryStream();
 XmlSerializer serializer = new XmlSerializer(obj.GetType());
 serializer.Serialize(temp,obj);

 temp.Seek(0,SeekOrigin.Begin);
 byte[] byteArray = new byte[temp.Length];
 int count = 0;
 while (count < temp.Length)
 {
  byteArray[count++] = Convert.ToByte(temp.ReadByte());
 }
 ASCIIEncoding asciiEncoding = new ASCIIEncoding();
 char[] charArray = new char[asciiEncoding.GetCharCount(byteArray, 0, count)];
 asciiEncoding.GetDecoder().GetChars(byteArray, 0, count, charArray, 0);
 return new string(charArray);
}
public static object DeSerializeObject(string s,object obj)
{
 ASCIIEncoding asciiEncoding = new ASCIIEncoding();
 byte[] inp = asciiEncoding.GetBytes(s);
 MemoryStream temp = new MemoryStream();
 XmlSerializer serializer = new XmlSerializer(obj.GetType());

 temp.Write(inp,0, inp.Length);
 temp.Seek(0,SeekOrigin.Begin);
 return serializer.Deserialize(temp);
}

Update: apparently XmlSerializer is pretty limited in which types it can reflect, it’s designed to fit current WDSL schema, which is limited itself. Well, not really, but in any case right now XmlSerializer can’t serialize multidimentional arrays, for example. In the code above you can replace XmlSerializer with SoapFormatter, while keeping everything else exactly the same. That will generate different (SOAP) envelope and you will be able to serialize any type of data.