Harry Li's Forum Nokia Blog

Batch testing in Symbian C++

kcomex — Mon, 15 Sep 2008 14:39:50 +0800

Hello folks, it's really a long time since my last post. Because of the busy schedule of my own company's project, I'm actually away from the Mozilla platform porting effort for a while. So it's a bit embarrased to carry out fresh experiences to share with you at this moment. However I will try to demostrate a testing trick which I learned from NSS (Network Security Service) testing section.

Probably all Unix gurus like the powerful shell script, and it is commonly used for testing purpose. In NSS the test cases are all called by several complex shell scripts, and there is a very flexibility to extend usage of these tests caller scripts. Unluckily we don't have a shell script executing environment on Symbian OS. But I was borrowing ideas of this testing method, by writing a console application and treat it as the main body to coordinate all test cases.

Here are some notes:

We use a config file to tell the tests caller which executables we want to call, their executable image names, the calling arguments, and expected return code.
There is a log file to log all activities about test cases calling time and return value status.
Frequently adjusted variables are defined at header as MACROs, like log file name and storage path. Please note STDIO_FILE macro should be defined the same as the standard output file in your C:\System\Data\config.ini
You could control test cases running mode, by add either pause before run every test or pause after every test case or both.

I hope this trick will help you on porting massive test cases project onto Symbian OS. And also if you have any comment on this, please feel free to send your comment.

Attachment 1 : source file, run_tests_symbian.cpp
Attachment 2 : example config file, tests.cfg

Cheers~ :p

Finishing the immigration to GNU make

kcomex — Fri, 01 Aug 2008 14:38:28 +0800

During last couple of weeks, I made the immigration of building Mozilla NSPR component with Carbide.C++ (still based on makemake and friends Perl scripts) to GNU make and autoconf friends. Here I will share some points I learned in this process.

First of all, if your open source project uses a lot of Writable Static Data (WSD), in theory even only one single variable, and they are used in dynamically linked library (DLL) target type, then your only choice is buying a RVCT compiler. Because GCCE compiler which ships with the S60 SDK has a defect that can not build a working DLL binary if there is WSD in it. This known bug is documented by Symbian already. Though in the document Symbian stated the problem is solved in a later GCCE release possibly, yet I had tried all GCCE releases which is later than 2005Q1C, then found some of them will generate an internal compiler error and some of them build the same defective binary almost the same as 2005Q1C. So I think I could give you the advice : try only GCCE compiler tools if it is shipped with Nokia or Symbian, otherwise you are spending your time on what has been proven already.

Then let's get back to the build tool chain immigration affair. Actually I observed all original makmake build process in Carbide.C++, and then try to copy all compiler and linker arguments to GNU make Makefile scripts. The work is not so hard, and the result is also convenient to verify. Here are some brief points need your attention:

Compiler/linker arguments are main additions feed to Makefile
Emulator builds and target device builds use different compiler/linker
Device builds will need post linker operations, elf2e32 is the tool we use
ActivePerl is used to calling the .RSS resource file compiling, other Perl probably won't work
Building statically linked library (.LIB target type) takes no post linker jobs to do, compared with .DLL and .EXE

Also, it really depends on the project scale you are working with. Like Mozilla NSPR component, it uses several config and make rules files to make targets for over two dozens operating systems, then finding which place is proper to add arguments takes longer than making the modify itself.

Finally, you could have a look at Bug 442706, the patch there will show you the modifications made against the original NSPR build scripts. A detailed description about how to build NSPR with GNU make/autoconf can be found at Mozilla Wiki.

Cheers~

Build Symbian C++ project with GNU make/autoconf

kcomex — Sun, 13 Jul 2008 13:09:39 +0800

Everyone who develops native Symbian C++ application knows that we use BLD.INF and *.MMP files to define a project. This is used since the very ancient Nokia 9200 SDK. The reason that Symbian uses this approach is quite similar with the reason Symbian made other unique features : because mobile devices have unique characteristics other than desktop computers. There is no UID for each appilcation on a desktop OS, also there is no compile time check on global writable static data use.

However, when porting open source project like Mozilla to Symbian OS, it becomes very impractical to stick with the original BLD.INF plus *.MMP solution. There are a number of reasons:

Portability :
Mozilla project is something we could treat it as a platform, framework or midware. So portability is something the community considers as the first requirement. The community maintains one code repository to generate binary executables for dozens of OS/CPU arch. combinations. So we can't say that you build everything except Symbian platform with GNU make while building Symbian targets with another tool chain.
Community compatibility :
We know the success of open source projects benefit from hundreds and thousands of developers around the world. The philosophy behind this doesn't mean we get software products without paying since they love contributing, it means they may be different experts on different sub-system. So we can't imagine that in future the FireFox for S60 has a security problem, a security expert has to learn how to build FireFox for S60 with BLD.INF and *.MMP for some time before he can get hands on the point of problem.
Large scale :
Because the code base and the number of components are quite large, it would be very difficult to write a BLD.INF to build the whole platform which you might noticed in my earlier blog. That's also a nature of the scale limitation of Symbian's own build system compared with GNU make. As a derived reason we have to use Mozilla's build system, the community have to run a server to build the whole source tree every several hours because with such a large code base almost every hour there are codes checking into the repository, then the old Symbian's build method will have to be modified heavily to suite this automatically build monitoring system.

The reasons we have to move from Symbian's BLD.INF/*.MMP build method to GNU make are stated quite clear above. Yet it's not a good idea to talk about how to port the system and dive into Makefile lines in a blog article because the blog post is really something more literature rather than technical codes . So I would like to give you two places to find the implementation and method behind all the immigration. First one is the [BUG 442706] where you could find detail patch about moving build system to GNU make/autoconf. The second one is Mozilla-Symbian project hosted on google-code which is a place for codes before they get checked into Mozilla main repository. With these two resources you could find solution of moving a Symbian project to GNU make/autoconf tool chain.

Limitation on Open C which matters HUGE project like Mozilla

kcomex — Tue, 03 Jun 2008 16:53:38 +0800

Open C is a quite powerful tool when porting open source projects to Symbian OS platform. But a mobile kitty is not a desktop monster, it doesn't have a so powerful kernel, especially you encountered some design limitations. For small projects which don't utilize lower or system level functions, you will wield Open C without any pain. However, projects as large as Mozilla will meet the limitations and avoid using incomplete or absent functions. So marking those limitations with good documentation and practical experiences is very important. Here I will try to tell you some of my experiences.

First of all, you need to read Open C documents carefully, Nokia and Symbian try to explain every limitation they know as clear as possible. So you don't need go discussion board and ask questions before check the documents. *BUT ACTUALLY*, the problem is not all about checking Open C documents, there are possibilities the problem is not caused by Open C but your code. In this case, the situation is not easy. You need practical experiences from others, by talking with colleagues or searching discussion board.

Here comes the body. The most incomplete group is libc Signal Handling, but what's lucky is we can find absence of these functions, you don't need to determine where the problem comes from, your code or Open C. The second hard one to over come is using posix_spawn() instead of fork() and exec(). Here I would give you a suggestion, read the complete fork() manual and compare the difference between posix_spawn(). Then check the context of your code, see if these differences applied. This technique is not so easy, even there are further hidden side effect, for example "In the child process created with popen3/popen/system/posix_spawn, operations on the inherited file may cause a panic." in P.I.P.S. V1.3 Release Notes. Bear in mind those side effects from practice, then the job would be less tough.

With two big stop showers ahead, I think the most important is not getting those limitations one by one, but a skill to find if the bug comes from _my_ code or Open C. Once the debug tricks are at hand, there won't be anything "won't fix"

Set a world record, with FireFox 3

kcomex — Thu, 29 May 2008 15:47:50 +0800

Wanna make a Guinness World Record? OK, I have two recommendations. First, balance thirty spoons on your face and second, download FireFox 3 on the day it get released . A more open web world with more players will benefit everyone, now let's wait and see what will happen the day FireFox 3 arrives.

Development Depends on Dependency

kcomex — Sun, 18 May 2008 16:00:55 +0800

It's quite a while since my last post. Although I'm not there, just because of the terrible earth quake happened in SiChuan China, I can't sleep well and concentrate on my codes while people there are suffering. God bless them, hope they could recover with time passes.

Actually this has been done two weeks ago. After evaluate the amount of porting works, we could go on with the dependency check of a specific project. For Mozilla, I had shown a framework picture:

From this picture we know the components in the lowest level are "NSPR" and Cairo". Because "Cairo" is only a graphics base, so we will start from NSPR first.

For other open source projects without such a components diagram, then how we could know its internal components dependency? I would sugguest taking a look at the build(configure && make) output, from this log everything is exposed even with any third party library used by that project. Anyway, large projects like Mozilla will take us more time to get everything clear. Also the more time we put on dependency check, the more clear we could get for a structural view, and the more easy for add or modify codes.

Evaluate the project before porting

kcomex — Thu, 08 May 2008 09:16:10 +0800

Porting an open source project is quite different from developing a new application for mobile phones. During past days without Open C, we have to rewrite lots of code to satisfy both Symbian OS rules/limitation and difference between mobile devices and desktop computers. Touching so much code will cost more development time, but what's worse is that making changes too much will drive you thinking "Should I start a new project rather than using only a little from this existing project?". So porting an existing application onto Symbian OS makes you keep concerning how much modification will you make, if too much consider starting a new one or just share the name or brand. As far as I know, HelixCommunity is the first big and famous open source project ported to Symbian OS, and it supports as early as v6.1. I checked out its source in 2004 (means their code nowadays might be different a lot), this is one example of making too much changes so you can tell few difference between porting like this or making a new project. Since S60 3rd Edition, there is Open C we can utilize, the job is easier. But still we can not ignore this consideration because of the nature of mobile phones. In one word, first step of porting an existing project is evaluate how much changes will be made approximately.

This document in Forum Nokia web site is very helpful, and generally speaking smaller projects are always easier to make decision if it could be ported than bigger ones. Smaller ones are always based on some existing framework, if the framework exists or can be ported on Symbian OS then the project itself will be fine in most cases. Huge project like Mozilla or Helix will give us an impression that it is likely impossible to port cause it has so many dependencies and self owned framework. However, we want eveidence to support our decision. Let's get back to Mozilla, after searching its website, there is an "C/C++ Portability Guide" which shows us an amazing likeness of Symbian OS programming limitations. From this article, we could learn Mozilla code will probably contain very little exceptions, RTTI and others that Symbian OS C++ does not support. Then with the help of Open C, we could have better confidence about porting without changing too much code.

Next we have to make deeper research on the inner structure of Mozilla platform, cause you can not start portting any piece of code without knowing which level it lies in and what it does actually. Stay tuned, I will bring more as the work goes on

Porting Mozilla onto Symbian OS

kcomex — Sun, 04 May 2008 01:08:44 +0800

Greetings Everybody! I have been a Forum Nokia Champion for over 1.5 years. Currently my major technology profession is developing native Symbian OS C++ applications mainly on S60, also I know some Python(PyS60), J2ME and Windows Mobile. Since last month I started concentrate my focus onto a very nice community, the famous Mozilla project. As I expressed before, I always feel extremely exciting about contributing my effort to something which will bring us openness, opportunities and freedom, just like the award winning desktop browser FireFox, and the possibility of running it on Symbian OS.

Frankly, at first I hadn't realized the size of the Mozilla Platform, until I read these from the book "Rapid Application Development with Mozilla":

It is 30 times larger than the Apache Web Server, 20 times larger than the Java 1.0 JDK/JRE sources, 5 times bigger than he standard Perl distribution, twice as big as the Linux kernel source, and nearly as large as the whole GNOME 2.0 desktop source—even when 150 standard GNOME applications are included.

Maybe the information from this book is a little out of date, also maybe size can not tell everything about an open source project. But what we could learn is that porting Mozilla platform onto Symbian OS will be a huge project which needs lots of hours, especially for those mobile developers with little Mozilla hacking experiences.

Here I'd like to share something I found in last 2 weeks digging, maybe some of you might become future contributor of Mozilla/FireFox on mobile devices. Also the reason I start my blogging about porting Mozilla onto Symbian OS is to continue the philosophy of open source software: making the development process and result product open, clear and collaborative.

It's important to know Mozilla well before get hands on code. Read their websites well, know more about open source, vision of Mozilla, historical event, maybe the unique culture from Mozilla people.
Mozilla platform is layered/modularized into several components, you could find the framework diagram in this blog.
Beginners should know where to find information or documents about Mozilla platform. There are two wikis which may confuse you: Mozilla Developer Center whose audience are mainly Mozilla platform technology users, clients, documents in this wiki are set to stone; Mozilla Wiki which shows articles in engineering stage, you could find useful information about "Mobile" in this wiki. There is another non-wiki portal with lots of well categorized documents, http://www.mozilla.org/developer/
Mozilla developers like IRC, meet them and feel free to ask your question at irc://irc.mozilla.org/mobile. If the link does not work for you, probably you'd like to have a FireFox with ChatZilla install.

There are a lot about porting Mozilla onto Symbian in the beginning days, I would like to write them down as time passes. I want to know more friends on this topic too, together we could make mobile Internet better with our efforts.

As part of my first blog, I want to say my thanks to Forum Nokia for providing us such a good chance to express our technology thoughts on mobile development. With this Forum Nokia Champion program I believe Forum Nokia community could be more and more popular, bring us more and more success stories one after another :)