SMS Character Set & Sending special charcters

Like they say necessity is the mother of invention, today, i was forced to understand the GSM SMS standards. I was trying to send a range of special characters in my message. After hours of grueling work and debugging, I realized only a handful of special characters are allowed.

So, I dug up the standard for SMS - GSM 03.38. This corresponds to an ISO character set called ISO 8859-1, which is extremely similar to Microsoft's Windows-1252 character set.

Sending a message through Kannel

I was using Kannel, the open source SMS Gateway software to send out the SMSs. Now, Kannel accepts all messages posted over HTTP only in the Windows-1252 encoding.

So if you're using ASP.NET, you must URL encode your text using the Windows-1252 encoding before making the HTTP request to Kannel. Otherwise, the message received on the device on the other end will look like gibberish.

Receiving a message through Kannel (Kannel Post)

When Kannel receives a message, It tries to see if the character encoding matches ISO 8859-1. If it decoding the message fails using the 8-bit character set, it tries 16-bit Unicode Big Endian (UTF-16BE).

If it is configured to post the message to a designated URL, it will first URL Encode the received text using using the determined formatting, and then supply the character set in the URL as a query string parameter.

If you want to receive your messages in ISO 8859-1, it is important to stick to the characters defined in the set. Failing to do so will call your Post URL with Unicode encoded text.

Resources / References:

• Kannel: Open Source WAP and SMS gateway
• GSM 03.38 Character Set: Ref 1, Ref 2, ISO 8859-1 Mapping
  
• ISO-8859-1 Encoding, Windows 1252 Encoding
  

How to hard reset a Microsoft Windows Mobile Smartphone

Well, I was trying to develop an applicaion to start on boot. When I changed the registry wrongly, the phone wouldnt' boot. The only way out was to hard reset the phone.

Officially, O2, a brand of HTC claims that hard reset can not be done by the end users. There is however a way to hard reset the Xphone / Xphone II / C500 /smt5600 / SP3 / SP3i / SP5 / SP5m.

Following are the hard reset instructions:

1. Turn off the phone. (If the power button fails to respon, take the battery out.)
   
   
2. Press the two soft keys while the phone is off. (I find it easy to set the phone on the table with the power button pointing towards the right. Then use the index and the middle finger of the left hand to press the soft keys simultaneously.
   
   
3. Press the power button for 1.5 seconds. (Timing is sensitive. You make take several tries to get it right. ) Essentially, the picture is that the phone is off, you are holding the two soft keys, and then you press the power button for 1.5 seconds and release it.
   
   
4. Press '0' when instructed. (When the phone will power up, and the screen will display an instruction to press '0' restore factory defaults, other keys to quit.)
   
   
5. Wait while the phone hard resets itslef.

References:

• Modaco Forum : Hard reset your smartphone
• PPCSG Forum
• Pocket PC Addict Foorum
  

Application unlocking a Windows Mobile 2003 / 5.0 Smartphone

Most Windows Mobile Smartphone devices by default are configured to be Application Locked. That is a rather irritating limitation for people like myself who develop on Windows Mobile devices quite extensively.

Since most Windows Mobile Smartphones are difficult to hard reset (especially mine - O2 Xphone II), a malicious app can make you run down to the service center to get your phone hard reset! Microsoft has built in the Application Lock for smartphone, to protect the device from malicious applications accessing the core windows privileged APIs.

What this means is that if you want to have functionalities in your application which interfaces to core phone functionalities, such as SMS message interception, keypad hooking, etc, you need to have privileged access (on locked devices) for your application. An application has privileged access to the core APIs if it is code signed by certain dedicated authorities:

• GeoTrust
• VeriSign

If you have a locked phone, and you want to run an application which is not code-signed, you must first unlock the phone. After digging through piles of segmented information, this is how I went about unlocking my application.

Download the following (bundle):

• Registry Editor (HTC Signed)
• Windows Mobile Smartphone Lock Checker
• RAPI Config & ResetSecurity.xml

Once you have downloaded the necessary components, perform the following.

Installation

• Connect your device to the PC using ActiveSync.
  
• Copy the registry editor, and lock checker to the phone memory through active sync file transfer
  

Check for application lock

• Use the device file explorer to navigate to the directory to which you had copied the lock checker.
• Tap the file to execute it.
• The application will tell you if you phone is locked. If your phone is unlocked, you can skip the rest of this article. Otherwise, go on to the next step.
  

Registry Updates

Change the value of the following registry keys in [HKEY_LOCAL_MACHINE \ Security \ Policies \ Policies]:

• Key: 00001001, Value: 1
• Key: 00001005, Value:40
• Key: 00001017, Value: 144

Running the Security Unlock

• Open command prompt
  
• Go to the path which contains RapiConfig.exe and ResetSecurity.xml
• Key in: rapiconfig /p resetsecurity.xml
• Hit Enter

Verify the Application Lock

• Use the device file explorer to navigate to the directory to which you had copied the lock checker.
• Tap the file to execute it.
• The application should now say that your device is unlocked.
  

Bibiliography:

• Modaco Forum: (Registry Editor Signed by HTC), (App unlocking guide)
• Usenet: microsoft.public.pocketpc.activesync (Guide to Certificate/Application Unlocking Windows Mobile 5 Smartphones)
• MSDN: A Practical Guide to the Smartphone Application Security and Code Signing Model for Developers (thanks Varun for the link).
  

Wondering how to hibernate when you have 1 GB + RAM?

And then you try to hibernate your laptop. The screen blackens out as if it is going into hibernation mode, but then it comes right back and windows throws you the message :

Insufficient system resources exist to complete the API

Have you got more than 1 GB or RAM? Well then, you've got a problem that you need a fix for. Microsoft Windows XP has a documented history of hibernation problem for computers that have more than 1 GB of main memory.

I recently got a Dell Inspiron 6400 laptop with 2 GB RAM, and somehow, from time to time, I couldn't hibernate. After doing some research on google, I stumbled upon the two articles on the Microsoft Knowledge Base:

• Article 330909 : Talks about the problem for the base Windows XP Professional, Tablet and Home editions. It provides a hotfix that can be downloaded and installed to rectify this problem. Unfortunately, this may not applicable to most of us since we are likely to be running service pack version of Windows XP.
  
  
• Article 909095 : Addresses the error messgae that is thrown when we try to hibernate. This applies to the latest releases of Microsoft Windows XP editions (SP2, Tablet 2005 and Media Center 2005). However, obtaining the hotfix for this one is not so straight forward.

The Knowledge Base Article 909095 stated clearly that the hotfix must be obtained from Microsoft support. So, I tried to find out how the support could be contacted, but it seems that there is a service charge for getting support.

The web site clearly states that the vendor could be contacted for this fix. So, I wrote a service request to Dell to get me the fix from Microsoft.

After juggling a few emails back and forth, the service person called me back saying that I must speak with Microsoft myself to obtain the fix. He joined me to a conference call with Microsoft Asia Pacific Support.

The entire process of the questionaire took about 20 minutes. But by the end of it, I had the fix in my mailbox. When I asked them why I must go through all this to get a simple fix, the support personnel explained that the fix is meant only for this certain purpose. If it is installed on PCs which don't have this specific problem, then, some other things may stop functioning!

The process does seem rather lengthy, but I'm extremely impressed with the quality of support Dell and Microsoft are providing. The problem has not surfaced since I installed the hotfix. Lets find out whether what I got in my mailbox yesterday was a proper cure!

Happy surfing with Opera & Proxomitron

Well... I have been a die-hard fan of Firefox, and its plug-ins. I still am, but I must say, getting back to Opera, years after I tried it out for the first time, has been a rather pleasant experience.

The particular things I liked the most about Firefox were the tabbed browsing, and the Adblock plug-in. They made browsing so much nicer by cutting the Ads & other tertiary items from the different pages. People have posted a lot of Adblock hosts and URL lists, and so, it was relatively easy to have an ad free experience online.

Nevertheless, I tried out Opera, just for the heck of it mostly. The Opera 8 interface I must say is fantastic. The browser loads fast itself, and it renders the pages rather nicely. The features which I liked the most are:

• Subscription to RSS Feeds
• Saving sessions, and restoring them later
• Zooming into a page, which scales entire page - including graphics and layout
• Mouse gestures

They made the whole browsing experience simply richer.

In spite of the cool features, what I missed the most from firefox to opera was the Adbock. Wherever I went, swarming ads took aeons to download, and once they downloaded, they distracted the reading.

So, after some googling, I came across Proxomitron, a personal ad-blocking proxy. The cool thing about Proxomitron is that it is absolutely transparent to all browsers / applications that need to use it.

I could use it from IE, Firefox, Opera, or any other application which allows HTTP requests to go through a proxy.

Among its features, Proxomitron has a set of rules built in which checks the HTTP header, and the contents for certain attributes. The attributes which can be set, or used for detection is rather extensive (daunting for first timers like me), but overall, it is quite easy to set up.

Once I configured Proxomitron on my computer, there was no truning back. It cut out almost ALL ads, and worked way better than Adblock. The built in pattern recognizers were much more efficient on Proxomitron, than they were in Adblock.

With the ads, being cut out at the request itself, the pages loaded much faster than they would have using adblock. It kept most of the ad placeholders intact, with small messages in red linking to the ads, so that I can go to the ad if I had wanted to.

The rendering kept the layout of the page intact, and I didn't have to loose sleep over a completely re-organized page.

My recommendation: Try it out. Its worth it!

Programming for Windows Mobile 2003 Smartphone

One of the primary reasons why I wanted to buy a Windows Smartphone was because it came with the .NET Compact Framework, and I could write programs for it!

Having worked with a Windows PDA a couple of times, it wasn't too difficult to get started. All you need are:

• Microsoft Visual Studio .NET
  
• Windows Mobile SDK
• A Windows Smartphone, or a smartphone emulator!

From the Visual Studio, you can create a Smart Device Application projct, and from there on, it is as simple as programming for windows forms.

When you run the application, you can debug it using the emulator, or the phone itself. In my first attemt at programming for smartphone, I created a notepad application called iNote.

iNote: Sample Windows Smartphone Notepad Application

Download: Executable| Source Code in C#.NET

References:

1. MSDN: Develop for Windows Mobile-based Smartphones Using the .NET Compact Framework
   
2. Microsoft Smartphone Programming
   
3. O'Reilly: Design Considerations for Microsoft Smartphone Applications
4. MSDN: Programming the Smartphone
5. MSDN: The .NET Compact Framework
   

Javascript Frame finder

What do you do when a page in a child frame is told to load a document in another frame? Rather straight forward I'd assume. You just make a hyperlink to that page, and set the target to the destination frame.

Now, the situation becomes a little more complex if you are using javascript. Especially if the script is being automatically triggered! Since in the javascript DOM, the frames are modeled in a tree strucutre, one can do an exhaustive search of all frames in the tree.

In the process of writing a function to traverse the entire tree to find a frame by its name, I realized that recurision is not possilbe in Javascript! (correct me if I'm wrong here :)

The script for searching a frame in a document is here :

var stack = new Array(100);

function FindFrame ( frameName )
{
 var frame;
 stack.push(top);

 while ((frame = stack.pop()) != null && complete == false)
 {
     // Make sure that the list of child frames is not empty

     var frm = frame.frames;
     if (frm == null || frm.length <= 0) return;

     for (i=0; i < rm.length; i++)
     {
         if (frm[i].name = frameName)
             return frm[i];
     
         stack.push(frm[i]);
     }
 }
 return null;
}

Reflect MS SQL Server Database Schema using SQL

Can a program learn the structure of the database? I guess thats what the code generators are all about. They use templates to automate the writing of redundant code.

Well... I decided to come up with something along the lines, but the first step was to learn the schema of the database that I was working with. After doing some research, I found out that there are a couple of ways of reading a database schema from a MS SQL Server database:

1. the Information_Schema view
2. the system tables in the database

Infomration schema is a convenient abstraction of the system tables, since it organizes the schma information for easy retrieval. Yet, there are pieces of information which I requried for my adventure which the information schema had ignored.

So, I had to go back to the system tables. Although it is not easy to get started with the sys* tables, they can be very powerful tools! I finally managed to get what I had wanted to using the system tables. Following is the list of scripts which can be used to retrieve schema information of a database.

List of databases on the server

USE MASTER

SELECT dbid [DatabaseID], name [DatabaseName]
FROM SYSDATABASES

List all the tables in a database

USE <database>

SELECT id [TableID], name [TableName]
FROM SYSOBJECTS
WHERE xtype = 'U'

List of all columns in the system

USE <database>

SELECT tab.id [TableID], tab.name [TableName],
col.colid [ColumnID], col.name [ColumnName],
col.type [DBTypeID], typ.name [DBTypeName], col.length [Size], col.isnullable [IsNullable]
FROM SYSCOLUMNS col

INNER JOIN SYSOBJECTS tab
ON col.id = tab.id
AND tab.xtype = 'U'

INNER JOIN SYSTYPES typ
ON col.type = typ.type
AND col.xtype = typ.xtype
and col.xusertype = typ.xusertype

List of all relationships in the system

USE <database>

SELECT DISTINCT
RO.name AS PrimaryTable,
RCOL.name AS PrimaryColumn,
FO.name AS ForeignTable,
FCOL.name AS ForeignColumn

FROM SYSFOREIGNKEYS FK

INNER JOIN SYSCOLUMNS FCOL
ON FK.fkeyid = FCOL.id
AND FK.fkey = FCOL.colid

INNER JOIN SYSCOLUMNS RCOL
ON FK.rkeyid = RCOL.id
AND FK.rkey = RCOL.colid

INNER JOIN SYSOBJECTS FO
ON FK.fkeyid = FO.id

INNER JOIN SYSOBJECTS RO
ON FK.rkeyid = RO.id

List of all Database Types

SELECT name [DBTypeName]
FROM SYSTYPES

Regular Expression Validators

Regular expressions are used to match patterns in a string. I suppose it is used quite commonly in Linux / Unix for 'grep' and other search tools.

Regular expressions could be used quite conveniently for checking text fields for ASP.NET Web Forms, or Windows forms.

ASP.NET comes with a Regular Expresssions Validator, which could be attached to a text box. Once the regular expression is specified, and the error message is set on the validator, then, upon the form submission, if the text box contains an invalid entry, the error will be displayed.

In .NET, the System.Text.RegularExpressions namespace has functions to perform regular expressions.

The Regular Expression Checker

Download: Executable| Source Code in C#.NET

I made a sample application called Regular Expression Checker which uses the .NET Regular Expression constructs.

The application applies a regular expression to a chunk of text, and displays the matches. I found it quite painful to build my application repeatedly to check whether my regular expressions were correct. This app saves me the time and the pain of constant building!

References:

1. A Tao of Regular Expressions
2. Regular-Expressions.info
3. Learning to use Regular Expressions