Kiran Patel's Forum Nokia Blog - Lost in the dark: Quality and consistency with Symbian Signed

Re: Quality vs quantity

burnsr — 2008-05-30T12:12:34Z

Paul,
It is not quite as simple as that as I am sure you are aware. If Symbian Signed removed the majority of testing from the criteria then the confidence of manufacturers and operators diminishes meaning that Symbian Signed would be less likely to be trusted by operators and the like. So, what they would then be likely to do is set up their own signing programme for their operator phones. This would then mean that there is likely to be fragmentation making it more difficult for developers to distribute applications.
Can you explain what the difference is with Microsoft compared with Symbian? Do you engage with Microsoft to gain some kind of trust level?

Quality vs quantity

Paul.Todd — 2008-05-30T11:53:01Z

Symbian Signed should provide trust the application has not been modified, not trying to provide a minimum bar for applications, the market will provide guidence on applications which are popular and which are not, Symbian, trying to distort the market will just lead to the people with the most money making average applications as they can afford to.

If Nokia wants to ship an application with the Nokia brand then they should have additional testing and quality standards defined by them which need to be met, but for the majority of developers who are not part of a big company the current model is fundementally broken. Symbian signed still miss some fundemental bugs during testing as it is impossible to predict what to test for all types of applications.

The general trend I see in enterprise software (at least) is that ISV's first try out an idea with Windows Mobile and then if it succeeds the the process of back porting to Symbian is started.
This is obviously not good for the Symbian platform as it means its largely dependant on Microsoft products to develop an application, rather than starting with Symbian and building applications.

Personally I wish Symbian would spend more time stopping people warez'ing developers software than hampering software development.

kiran10182 — 2008-05-30T09:23:20Z

Thanks Amit and Nigel for your comments,

I wish all these comments are considered in positive terms and we get something useful in near future.

nigel.brown — 2008-05-29T16:59:48Z

Yes, in my view there is something slightly wrong with the whole concept.
The way it is all set up the trust relationship is between the Symbian signed and the code. It should be between Symbian signed and the developer.

You can't trust code!

E.g. You write a Trojan which passes all the tests. So, Symbian say they trust this code and allow it on their handsets. Then on July the 4th it deletes all the files it can get its hands on.
It isn't the code that gets slapped in irons and hauled off to a small room at the bottom of the NRC. It is the developer.

You can trust the developers!

They have something to lose. They should be trusted (or not).

But, it is just an opinion.

amitkankani — 2008-05-29T07:40:26Z

Well said, and I agree that signing should be made easier to encourage people.

My key point would be:
It should not be the application quality/bugs which should be tested, but instead, any rules being broken, which could affect customers device should be tracked. The quality of application is always left to the developer... ain't i right ?

Re: Wifi phones

kiran10182 — 2008-05-28T16:53:20Z

But can you check it in .pkg file while installing your application? In such cases it should be taken for granted because we do not have proper cases/documentation to check such things. Let me repeat my words, it should be motivating rather than discouraging.

Wifi phones

burnsr — 2008-05-28T16:44:44Z

The difficulty with something like wifi is that it is really up to Nokia to provide the app developer with a code to ask if Wifi is supported on the phone or not (maybe they do, I don't know). This way it is easy for you to display a message to the user saying the phone does not support Wifi and therefore the app will not run.

Re: Testing

kiran10182 — 2008-05-28T15:54:48Z

Of course, i would like to describe that. For e.g: 1) You are getting failure for UNI-10 Scalable UI compliance for the wi-fi issue. Is it transparent?

2) You send a waiver and you get answer. "Your waiver is rejected". But why? Is it transparent?

And again i am saying that i am not against Symbian signing process. And i believe in your first paragraph. And i do appreciate your wordings "There needs to be a stressed usage test in order to give a level of confidence that the application is of some level of quality". And i hope it should be implemented.

Testing

burnsr — 2008-05-28T15:40:51Z

Testing can never be 100% consistent between different Test Houses because the testers are human and don't all follow exactly the same steps. There needs to be a stressed usage test in order to give a level of confidence that the application is of some level of quality. As pointed out before certification is for such a wide range of applications how do you ensure consistency is 100% or close. It seems pretty difficult.

>"All in all it should be transparent process with good reasons for failing your application."
Can you describe how the process is not transparent and how you suggest it could improve?

kiran10182 — 2008-05-28T13:46:01Z

That is my question. Why should you face "plenty bizzare" bugs!!! Why it works in one test house and why not on another! And just for the sake of experience, one would really not like to spend hundreds of Euros, would you?

Re: Lost in the dark: Quality and consistency with Symbian Signed

joukovierumaki — 2008-05-28T13:30:44Z

Sometimes it helps to resubmit your app to another testhouse, as everyone does the testing differently ;)

We have had apps that have passed on testhouse several times without any problems and then next time when submitting it into another one (for fast track or another reason) - there will be plenty of bizarre "bugs" found. Usually lot of them are withdrawn after we ask from them what werethey thinking.

kiran10182 — 2008-05-27T20:32:47Z

If you have been using the same test house for your applications then in some percentage that should matter. And i must admit that it should be neutral enough as it's a certification.

By the way one should not be charged for resubmission fees only for some minor stupid failures. Big company can afford this but where would Freelancer go!!! All in all it should be transparent process with good reasons for failing your application.

Is it possible?

Sorcery-ltd — 2008-05-27T20:22:19Z

Is it possible to have:
1. Clear and definitely without any interpretation test cases.
Without also having many hundreds of test cases? The current test cases have masses of room for interpretation which allows the criteria to be simple. I'm not even sure it's possible to write "one size fits all" test criteria for diverse applications. This is only likely to get worse as various port enablers are added and we get more applications ported from other platforms.

Also, I think if you want:
2. Strong and motivated test reports.
Then you're going to have to pay more to get better test engineers certifying your application.

In an ideal world, the test houses should completely ignore who is submitting the application and judge it purely on it's merits - it is a certification after all. In reality I'm sure that if you have a well known company and they've used a test house several times then their certifications are going to get smoother.

Mark

kiran10182 — 2008-05-27T19:38:25Z

Thanks for your comments. Yes you are right with your key points. And moreover other aspects should be put under consideration. For e.g: Company's reputation in market, history of company, company's contribution in overall symbian sign program etc...

Excellent analyze.

StephBel — 2008-05-27T19:09:35Z

Your point of view matches exactly with mime.
Code Signing is necessary and Symbian Signed as always keep improving, especially with the Express Signed.
Symbian Signed Test Criteria looks much more precise but the Test House results are always some kind of surprise.
I see two “key points” to improve the process:
1. Clear and definitely without any interpretation test cases.
2. Strong and motivated test reports.