Paul Todd's Forum Nokia Blog

Variants and Symbian

Paul.Todd — Wed, 18 Jul 2007 23:08:35 +0300

Been a while since I had a chance to post. What with the startup, summer holidays and work intruding.
The Champions day in Singapore was good fun, even got to meet Jukka and Anthony Pranata and
even got to do a day out around Singapore with Sammi and PushL.

During the week, I had an interesting chat with Kevin Sharp about articles and blogs, so there
will be more source and articles on the Wiki and the blog will cover more reasons about why
I did something a certain way, rather than trying to cram it into a blog entry

Recently I posted a new Wiki article which implements a Variant class for Symbian.

A lot of the problems in Symbian revolve around the ownership of objects.

In particular my implementation differed from Nokia's one in that I decided that if the user wanted to
add a string to the Variant then the Variant would own that string or data.

This differs from the approach Nokia took with their variant class in the AIW* API.

I decided amongst three possible types.

Use a T class and push the ownership issues onto the person using the class. For example if you had an
HBufC then you were responsible for the resource management which is a bit of a cop out really and not
aligned with the standard Symbian programming idioms.

Use a C class then the resource management issues are managable using Symbian Idioms, but its a pain to always
have to dereference a pointer.

Finally I decided on an R class as the halfway point between an T class and a C class. The variant can own
resources, so if you have dynamic data, then you need to put the variant onto the cleanup stack or call Release
when you have finished with it. This also means you can treat variants more like local variables, rather than
pointers.

Internally the variant stores all the classes it supports as a union, much like the Windows implementation does,
though not as flexibly, primarily becuase Windows handles arrays so much better than Symbian. (Though Symbian
could make a much better job of it if the RArrayBase class were publically inherited!)

What else do I want to do with RVariant?
I was thinking about adding TPtr support; the problem being when it comes to serializing the data, where you do not
know the destination, so if you are going across processes or writing to disk you invalidate the pointer.

One thing is also adding in a TAny* pointer so arbitrary pointers could be put into a variant.

With both of there the role of ownership really needs to be considered and for that reason I decided against putting
any specific Variant type for pointers, at least until I get the role of ownership sorted out in my head!

One thing that might be useful is a change type function. In Windows there is a function called VariantChangeType that converts a variant from one type to another type where possible.

Use and abuse of Symbian idioms

Paul.Todd — Mon, 02 Apr 2007 23:18:55 +0300

I saw a very interesting snipped of code on the newsgroups the other day. It showed how people can so easily misuse Symbian programming idioms if they are not careful

    RArray<TInt>* array = new (ELeave) RArray<TInt>;
    array->Append(1);
    array->Append(2);
    delete array;

This is not the original class, the original class used another R class but the problem is still the same.
To all intents and purposes this look like safe c++ code but the coder failed to realize that an RClose MUST be closed and cannot be deleted.

I wonder how many other people have been bitten by this issue?

Some tips on C classes

Paul.Todd — Sun, 25 Feb 2007 21:51:51 +0200

Here are some reminders about C type classes for a typical question I saw on NewLC

Why does my C class not initialize the members to zero? Is this a bug?

1. Do not create C classes on the stack.
This is the reason the member variables do not get set to zero.

2. C classes allocate dynamic memory.
This will result in leaks if any of the methods following leave.

3. C classes almost always do two phase construction.
This is where the resources are allocated and at this point can leak if the object is not on the cleanup stack

4. R classes allocate handles.
- These handles may be leaked if any of the methods following the Open method leave.
Be very careful if you open a handle and do not put it on the cleanup stack!
People put returns in the middle of the function and forget to handle functions that leave.

5. Programming standards in Symbian are paramount.
-If the class allocates memory, derive it from CBase and put it on the cleanup stack.
-If it is using handles, use an R class and put it on the cleanup stack
See the Symbian Wiki for more information on standards

6. Follow Scott Meyers advice and make your constructors and ConstructL class private.
-So that the class cannot be inadvertantly created on the stack.
-Make you copy constructor and assignment operator private and unimplemented so you get compile errors if you accidentally use them.
-Always use NewLC or NewLC to force people to use the proper constructor
Effective C++: 50 Specific Ways to Improve your Programs and Design

7. Put the address of the item you expect to be at the top of the stack in the argument for PopAndDestroy
There are overloads on PopAndDestroy that allow you to provide a pointer for the item you expect to be on the top of the stack after popping and destroying items.
You will get a panic if this item is not at the top of the stack after the items have been popped.

CStack and a little problem

Paul.Todd — Tue, 14 Nov 2006 22:35:08 +0200

Someone emailed me a while back on how to use CStack. A quick search of the devkit with Copernic search showed no examples so I decided to post some code on how to use it.

The Symbian implemenent of a stack is a simple array where items are appended to the end of the array. This is both simple, elegant and unobtrusive.

First decide if the stack is to hold pointer or structures.
If the array is to hold pointers, then does the array own the pointer? Normally the answer is yes.

Lets say we want to create a stack of strings, then these strings would be implemented as HBufC's
so to make life easier, when a string is on the stack, the stack owns the the string.

So the stack can now be declared as (note the stack owns the data it contains!):
CStack<HBufC, ETrue>* myStack = new (ELeave) CStack<HBufC, ETrue>;

As we have declared it on the heap, we need to remember to push it onto the cleanup stack
so that if there is a leave it get cleans up nicely.
CleanupStack::PushL(myStack);

To add an item onto the stack is simple
_LIT(KFirstItemOntoTheStack, "My First Item");
myStack->PushL(KFirstItemOntoTheStack().AllocL());

Here we create a copy of KFirstItemOntoTheStack and then put it onto the stack, note
that at all times it is leave safe and nothing leaks if for example PushL fails.
This is because when we created the stack class we told it that it owns all the memory

Of course now we can get the top item, which returns an HBufC*
HBufC* top = myStack->Head();

and check to see if the stack is empty

if (myStack->IsEmpty())
       {
       // do something
       }

There are also the usual Pop to remove the top item (but not delete it!) and Head to get the top item,
which for some reason is not declared as Top, like my CS 101 class taught, go figure?

Now we have finished with the stack, we can delete it, this also deletes all the items
owned by the cleanup stack.
CleanupStack::PopAndDestroy(myStack);

This is a pretty good implementation and to be be quite honest the only thing I would really like to see
would be for it to be an R class derived from RPointerArray.

Now for a little quiz:
What is wrong with this code?

struct TMyVersionStruct
{
    TUint32        iVersion;
    TUint32        iItem1;
    TUint32        iItem2;
    TUint32        iItem3;
    TUint32        iItem4;
    TUint32        iItem5;
    TUint32        iItem6;
};

LOCAL_C void TestCrash()
    {
    // The production code reads a blob from disk and it looks correct
    // in the debugger but it itermittantly crashes,
    // here however we simulate what the read from disk as the problem
    // is not in the reading...

    HBufC8* item = HBufC8::NewL(sizeof(TMyVersionStruct));
    item->Des().SetLength(item->Des().MaxLength());

    TMyVersionStruct version;
    TPtr8 ver((TUint8*)&version, sizeof(version));
    ver = *item;

    delete item;

   // do something with version
    }

Its an interesting if very subtle problem that seems to come up when memory is very fragmented and is
fiendishly annoying.

Answers next week.....

Shutting down Servers and Threads Part 2

Paul.Todd — Tue, 19 Sep 2006 17:43:30 +0300

In the first part we covered how to correctly and cleanly shutdown a process.

Now we are going to cover how to shutdown a thread.

Typically people start threads to do some form of processing but often fail to plan how to shutdown a thread.
I often heard the old saying "Fail to Plan then plan to fail". It is essential to decide early on what conditions are necessary for the thread to stop.

By terminating threads all sorts of issues can be introduced or can can occur depending on when the thread is killed so it really makes sense to plan how to end the threads aside from using terminate or kill.

There are a number of ways to stop a thread and we are going to cover the most important ones here

Watching for a thread to die
In Symbian the easiest way to watch for the thread dying is to use the RThread::Logon method.

You can use it as follows:
RThread thread;
thread.Open(myThreadId);
TRequestStatus watcher;
thread.Logon(watcher);

// start your thread shutdown here
// The below will block until the thread dies
// so make sure you shutdown quickly, otherwise you will
// need to use a wait dialog or something similar
User::WaitForRequest(watcher);

Note that until the thread dies, the thread that is executing the logon/WaitForRequest will be blocked, though you can encapsulate the logon inside an active object. In practice the time from the point where you request the thread to when it is shutdown is sub-second so its normally not worth the hassle.

Using a TRequestStatus to stop a thread
This is normally the "Symbian" preferred method.

Here you pass a pointer an TRequestStatus the thread and when you want to shut down the thread you just complete the TRequeststatus and wait for the thread to complete. Note that the TRequestStatus is "used" by the thread being shutdown and not be the calling thread. It looks intuitively wrong but works.

TRequestStatus* killer = NULL;
mythread.Create(_L("Thread"), MyThreadFunction, 1000,1000,2000, killer);
....
.... we now want to stop the thread
TRequestStatus waiter;
myThread.Logon(waiter); // watch the thread
User::RequestComplete(killer, KErrNone); // tell the thread to stop
User::WaitForRequest(waiter); // when it closes the wait will end

LOCAL_C void MyThreadFunction(TAny* aAny)
{
   // setup stuff like active scheduler etc.

   TRequestStatus threadstop = KRequestPending;
   TRequestStatus* sink = REINTERPRET_CAST(TRequestStatus*,aAny);
   sink = &threadstop;

   // do something e.g.
   TRequestStatus event;
   wsSession.EventReady(&event);
   // Note WaitForRequest allows two AO's!
   User::WaitForRequest(event, threadstop);

   if (threadstop != KRequestPending)
       {
          // this is called when the stop TRS was
          // signalled by the main app
          //
          // do what ever you need to stop the thread
          // e.g. shut handles, delete object etc.
          return;
       }

   // event was signalled so do some more processing
   // this was not the thread stop TRS so we can still run
   // as we have not been requested to shut down
   }

Using a global variable
In your main application you can create a global variable, say static TInt32 gShutdownFlag = 0. Of course it need to be static, you could pass a pointer across just as easily.

You can then set this from your main application and in your thread loop you can check to see if this flag is correctly. this is particularly suited to games as these generally do not use Active Objects heavily.

A source of concern might be that reading and writing data across threads is unsafe. Generally speaking read and writes are safe provided they are 32 bits wide. However in OS 9.x Symbian have introduced a new API in the User module, called SafeInc and SafeDec which guarantee that this is atomic.

Another method which I have'nt used is the Publish and Subscribe API. This API allows different processes to work with a common system wide global variable.

Microsoft CE Emulator Goes Open Source

Paul.Todd — Thu, 20 Jul 2006 11:28:20 +0300

Microsoft are definalty leading the way with their Windows CE Emulation (or to be correct Simulation).

Features include

CPU emulator that executes the ARM instruction set by JIT-compiling to x86

An MMU emulator to support virtual memory and page protection

A motherboard emulator that contains emulated RAM and NOR flash memory

A collection of peripheral devices attached to the motherboard: serial ports, LCD controller, touchscreen, keyboard, interrupt controller, programmable timers, real-time-clock, network cards, audio, etc.

A “DMA” interface which allows a Win32 application running outside the emulator to communicate with a WinCE application running inside the emulator, using a simple socket-like programming model.

Even though this is in effect a techview release of the emulator, it would be really nice to see Symbian provide the same level of emulation that Microsoft do, especially regarding the native arm support which provides full ARM emulation including alignment faults, which are the bane of every project that has one hacker on it :)

Come on Symbian, give us a better emulator! If Microsoft can do it and make it open source, this definitely raises the bar for you guys!

p.s FileMan is stuck in Symbian signed :(

SyExpat Port to OS 9.x

Paul.Todd — Wed, 21 Jun 2006 21:12:44 +0300

I was going to post this on Sunday but with the uncharacteristically good British Summer and England playing Sweden
in football, this slipped over to Wednesday.

Originally I thought that porting SyExpat to OS 9.x would involve a large number of changes.
In the end this turned out to be not the case.

The simplest fix was to xmlparse.c. It seems that the expat guys mixed up the return code enum and the error
code enum, so changing it is all that is needed for the main expat source code.

A slightly tricker problem was the with GCCE. It seems that it likes to define XMLCALL to something. GCCE seems to think it supports Microsoft extensions but it only does it in some cases.I think it was _MSC_EXTENSIONS, so in order to fix this we need to make sure that expat_config.h is defined before expat.h is included in the source files. This will then define XMLCALL and so stop expat_external.h defining it to the default.

The only change then resolves to just make the SyExpat.cpp file include the expat_config file before expat.h
as this sets up a number of defines that specify how the functions are to be exported and so this sorts out the export declarations not being handles correctly.

Hence we just define XML call as blank and this fixes the compiler issues. See expat_config.h for more
information.

This is now sufficient to be able to build SyExpat for OS 9.x and still retain backward compatibility.

Unfortunately I have run into rather a nasty issue with Carbide.vs and the standard C library.
The whole sorry tale is here - http://discussion.forum.nokia.com/forum/showthread.php?t=75575 so for the
moment it seems there are some serious OS9 issues to fix first, even in the maintentance release.

Thanks to Harri Salminen for his fix for name space parsing which has now bee incorporated as well.

If anyone has any specific problem areas in Symbian they would like covered please email me and I will try to knock something up to show it is supposed to work.

Next week I will look at how to use MobInfo (and CTelephony) as a number of people seem to have a problem understanding how to use active objects with these libraries.

Addendum:
I have just found out I cannot attach files to blog entries so I will put the actual revised code up on my website.

RArray and Resource Mangement

Paul.Todd — Mon, 12 Jun 2006 00:37:11 +0300

If you are not using RArray, you should be!
It offers a substantial performance improvement over the CArray.. implementations for almost all the cases I can think of.

It is fast - RArray is coded in ARM assembly in release versions of the code.
It does not use asserts internally and also does not leave, it just returns error codes, so the use of trap statements in runtime significant areas of the code can be minimized or reduced.
It is lightweight
It is a wrapper over a more complex implementation of a generic array.
There are templated version of TInt and TUint built into EUSER so it does not bloat your executable.

Most of RArray is fairly simple to understand and works the way any generic array implementation works.

The biggest problem I do have with RArray is that the search and insert functions do not allow a variable to be passed to them so you are required to use a global or some other hack to get a state variable over into the comparison function.

The areas I am going look at are how RArray manages resources and how RArray can be applied effectively used to improve performance.

Before we can continue there are some questions that need to be answered:

How does RArray allocates memory?
If you play around with RArray you will soon find out that it allocates memory via the User::Alloc functions and these allocations are always a multiple of 4 bytes.

What RArray Constructors are there?
There is a default constructor, a constructor that specifies by how much the array will be expanded when it runs out of spare slots, and most importantly a constructor that maps onto a memory locations- but more of that later.

What is not present is a copy constructor, so what will happen then is that the C++ compiler will create a default constructor that is a bitwise copy of the item it is being copied from.

By default RArray does not have any items, however if one item is added, it preallocates by default 8 items, well okay KGranularity items so it can avoid memory fragmentation when adding items by avoiding a reallocate and copy each time an item is added.

Resource Management:
For the purposes of the example we will create a structure called TVector which has two integers in it.
THIS STRUCTURE IS A MUTIPLE OF 4 BYTES WHICH IS MANDATORY FOR RARRAY ELEMENTS!

struct TVector
{
TInt iSpeed;
TInt iDirection;
};
typedef RArray<TVector> RVectorArray; // Note that we keep the R prefix on our typedef

(1) static const TVector data[] = { {1,1}, {2,2}, {3,3} };
(2) const RVectorArray array(sizeof(TVector), CONST_CAST(TVector*,&data[0]), sizeof(data) / sizeof(data[0]));
(3) RVectorArray array1;
(4) array1 = array;
(5) TInt count = array.Count();

Each of the lines is numbered and documented as follows:
(1) we create a static const memory block. This could also be a block of memory from somewhere else such as a scanline or bitmap row.
(2) We now create an RArray that maps that memory into the RArray, note that the array is const so we can't expand it by accident.

It is important to read the documentation on this as it is very clear that this constructor does not allocate any memory nor take ownership of the memory, it just provides a mapping for developers to use the RArray with blocks of memory that needs to be manipulated.

(3) We now create a default RArray which has not memory allocated and it does not have any elements.

(4) Finally we create a clone of the array. It is vital to note that the compiler has underneath generated a default copy constructor and created a bitwise clone of data into array.

(5) Finally get the number of items in array after we have cloned it. What is the value of count?
Answer: See below

IMPORTANT NOTES:
(2) This just creates a mapping of the memory structure into the RArray is does not force the RArray to allocate any memory, hence there is nothing that needs to be put onto the cleanup stack. This is a very important point that a lot of people get wrong.
(4) This creates a clone of the array but ownership of the items in the data array is not taken
(5) Since array is a clone of data, count is 3
(6) When the function goes out of scope the arrays are destroyed, but we do not call reset. This is very important, because the arrays do not own any memory they MUST NOT be freed, so calling Reset is a big no-no! Likewise putting it on the cleanup stack is also a big mistake and will cause crashes.

Resource Management Part 2:
As I mentioned earlier, the underyling memory allocator for RArray is User::Alloc, so providing we allocate memory via this mechanism, we can move memory blobs from our structure into an RArray and then use the RArray.

(1) TVector* data2 = REINTERPRET_CAST(TVector*, User::AllocL(sizeof(data))); // actually sizeof(data) / (sizeof(data) / sizeof(data[0]))
(2) Mem::Copy(data2, data, sizeof(data));
(3) RVectorArray array2(sizeof(TVector), data2, sizeof(data) / (sizeof(data) / sizeof(data[0])));
(4) CleanupClosePushL(array2);
(5) const TVector element = {4,4};
(6) array2.Append(element);
(7) count = array2.Count();
(8) CleanupStack::PopAndDestroy(&array2);

Each of the lines is nummered and documented as follows:
(1) We allocate a block of memory using User::Alloc that is size of the 'data' structure (3 * sizeof(TVector))
This data is not put onto the cleanup stack, though of course we could put it onto the cleanup stack here and then remove it from the cleanup stack on line (4) and add the RArray back to the cleanup stack.
(2) We now copy the data from the const memory structure to our newly allocated memory blob
(3) We now create an RArray that maps the memory we just allocated to the internals of an RArray.
(4) Because we allocated the memory somewhere else, I decided that the RArray should now take ownership of the array, so that if any of the following lines should leave the memory will get correctly cleaned up by the RArray.
(5) we declare a new element
(6) This element is now added to our array that owns the memory. Where the array is full, the array will be expanded to cater for the new element, which may result in a memory reallocation, hence BE VERY CAREFUL where ownership of memory blocks is made.
(7) What should count now report? Answer: 4
(8) As we transfered ownship of the memory block to the array2 RArray, the cleanup stack may now delete it. If we did not put it onto the cleanup stack, we can just call Reset which has the same effect.

IMPORTANT NOTES:

(4) Is very important as you want to move ownership of the memory block to the RArray as soon as possible, especially if you want to add or remove items from the RArray.
(6) We should really check the return value from the append as it could have returned a negative value which would indicate that the item failed to expand. I have seen a number of cases where the error code is never checked and so unexpected things start happening.
(8) We do not neccessarily need to put the array2 as a parmeter to the PopAndDestroy but it provides an additional check that the item at the top of the stack is the item we expected it to be.

Some other things about RArray and RPointerArray

This is an advanced example of how to use resource management and RArray. Its not for beginners.
In real life I have used to read a block of integers (5000) from disk as a TPtrC8 and then cast it and move it to the RArray where the RArray takes ownership of the data. Which knocked about 10 seconds off the load time.
The arrays only take items that are 4 bytes or a multiple of 4 bytes, so if your object is say 9 bytes then the RArray will internally allocate 12 bytes for each item, hence take care when creating structures. Likewise, if your structure has different size items in it, check at compile time that the size is what you expect it to be. I have encountered situations where the sizeof has reported a different to what I had expected becuase of alignment issues in structure fields.
To get the address of the block of memory allocated to the RArray, take the address of element zero.
i.e. TVector* ptr = &(array[0]);
RPointerArray does not have an implementation that destroys all the pointer objects. Normally you would do this via ResetAndDestroy. For some reason Symbian decided to put the implementation of this in mmfcontrollerpluginresolver.h. The function is called CleanupResetAndDestroyPushL
One source of problems with RArray is when appending a large number of items to the array in one go.
For example lets say:
(1) RArray<TInt> array;
(2) for (TInt i=0; i < 1000; i++)
(3) array.Append(i);

Each time (3) is called when the array hits capacity, it will need to be expanded by its default granularity (8) which means that there will be a large number (124 reallocations) during this example.
In OS 9.x Symbian has added a Reserve function which preallocates a specified number of items in the RArray so that when adding items, the array does not need to be expanded which reduces memory fragmentation.
To simulate this in OS7 and OS8, we can reuse the tech above to allocate and transfer ownership.

For example:
(1) RArray<TVector> temp(sizeof(TVector), REINTERPRET_CAST(TVector*, User::AllocL(1000 * sizeof(TVector))), 1000);
(2) for (TInt i= 1000-1; i >= 0; i--)
(3) temp.Remove(i);
(4) RArray<TVector> array3 = temp;

Important Notes
(1) We allocate 1000 items and put it into the RArray.
(2,3) We then remove the items starting from the back which prevents horrific memory copying.
(4) Finally assign the new array to the existing array. Note that if the existing array (array3) already has items then you will need to call Reset on it before assigning the items, otherwise the memory in the array will be orphaned and you will get a memory leak.

In Conclusion:
I hope this has provided an example of how to use resources in RArray where speed is critical. Needless to say, make sure that if you use this technique you keep it localized to one function so that the maintainer can know exactly what you are doing as these techniques need to be used to extreme care and only where there is time critical code.

Next week I want to discuss how I ported SyExpat to OS 9 which I have been promising to deliver for ages now. Its coded, I just hav'nt had a chance to document what I changed and why.

I only just posted a comment about multidimensional RArrays so I might publish some additional notes about these this week as well.