Numberz Challenge

Last week I wrote about test design and imagined an app where I thought automation could help with testing. I had some time this morning, so I went ahead and made the application.

image

When you press the “Roll!” button, the app generates 5 random numbers between 0 & 9 (inclusive), and sums the numbers in the “Total” above. The stakeholder’s primary objectives are that the numbers are random, and that the summing function is correct.

To be honest, this is an app where you could probably prove functional correctness via code review, but let’s stick with black box testing for this exercise. The hints in the original post may or may not provide clues to defects in this implementation (that may or may not exist).

Here’s the challenge

If you want to play a little game for me and do some testing, test this app and tell me if it’s ready to ship (according to the stakeholder expectations). Report your findings in the comments section by 11:59pm on Wednesday, September 7. For the best report (tie-breakers go to earlier submissions), I’ll send you a copy of HWTSAM (optionally autographed), or a $20 Amazon gift certificate.

In a perfect world, I’d have an object model or some other way to test the functionality without using the GUI. Sorry – for this example, the world isn’t perfect.

Oh yeah – the app is Windows only – sorry. But if someone wants to send me a Mac Air, I promise I will write Mac version of the app.

I’ll answer any other questions on twitter (@alanpage), or in the comments. Good Luck.

If you want to play, a zip file with the binary is here.

Comments

  1. It is *not* ready to ship.

    I’d send the bitmaps if it were possible.

    Steps:
    -hit Roll
    -record each result in Excel and calculate the Sum independently.
    – validate Sum equates to Numberz’ sum
    -Repeat until it doesn’t.

    RESULTS:
    Numberz told me that:
    1) 9+3+9+7+1 = 33 (actual: 29)
    2) 5+8+4+6+0 = 22 (actual: 23)
    3) 6+3+4+8+6 = 24 (actual: 27)

    In Thirty-Two iterations, I encountered 4 summation failures.

    Distribution of the sums looks nice (bell curve). However, the number 2 is showing up only half as often as the other numbers. (with 32 iterations, though, this is a far cry from being proof of a problem)

    1. Excellent Brent. The “developer” has now dropped a new version of the app (the link above is updated).

      This is all part of the simulation – no other updates are planned for the exercise.

  2. OK, I got no confirmation on my previous post (and I don’t see it here) so I’ll try again. Sorry if it’s ‘double’ now.

    I see from the earlier comment that some person got to test the app. I on the other hand can’t even start it. It says that my computer is missing MSVCR100.dll and that I might wanna try reinstalling it….

    1. I’ve added msvcr100.dll to the zip package (the exe is unchanged).

      Oh – and the first reply was sent to spam – I marked it as not-spam so hopefully the engine will do better in the future. Your ID should be auto-approved for comments now.

  3. Ok thanks, now it’s running

    Found one issue though. The X mark on the right upper corner doesn’t work correctly. It should close the application when clicked on but it doesn’t. It only works when I use a right click on the title bar (showing a popup menu with options ‘Move’ and ‘Close’) first and then click the X. The ‘Close’ option in the right click menu works correctly as well by the way.

    In case it’s an OS specific issue…I’m using Windows 7

  4. 2%: -1

    Sample:

    27 8 5 8 3 2 -1

    Automation code:

    using System;
    using System.Collections.Generic;
    using System.Linq;
    using System.Text;
    using System.Windows.Automation;
    using System.IO;

    namespace RollTest
    {
    class NumberzTest
    {
    AutomationElement numberzWindow = null; //Main UI Window element
    //The following ID can be obtained from tool: UI Spy
    string btnRoll = “1000”; //ID for Text element of output window
    string text1 = “1002”;
    string text2 = “1003”;
    string text3 = “1004”;
    string text4 = “1005”;
    string text5 = “1006”;
    string textTotal = “1007”;

    static void Main(string[] args)
    {
    NumberzTest autoClient = new NumberzTest();
    //Create callback for new Window open event. Test should run only when the main Window shows.
    AutomationEventHandler eventHandler = new AutomationEventHandler(autoClient.OnWindowOpenOrClose);
    //Attach the event with desktop element and start listening.
    Automation.AddAutomationEventHandler(WindowPattern.WindowOpenedEvent, AutomationElement.RootElement, TreeScope.Children, eventHandler);
    //Start caculator. When new window opens, the new window open event should fire.
    System.Diagnostics.Process.Start(@”D:\lixiongUSB\Numberz\numberz.exe”);
    //Wait execution
    Console.ReadLine();
    }

    void OnWindowOpenOrClose(object src, AutomationEventArgs e)
    {
    if (e.EventId != WindowPattern.WindowOpenedEvent)
    {
    return;
    }

    AutomationElement sourceElement;
    try
    {
    sourceElement = src as AutomationElement;
    //Check the event source is caculator or not.
    //In production code, string should be read from resource to support localization testing.
    if (sourceElement.Current.Name == “Numberz”)
    {
    numberzWindow = sourceElement;
    }
    }
    catch (ElementNotAvailableException)
    {
    return;
    }
    //Start testing
    while(true)
    {
    ExecuteTest();
    }
    }

    void ExecuteTest()
    {
    //Invoke ExecuteButtonInvoke function to click buttons
    ExecuteButtonInvoke(btnRoll);
    var ret=GetResult();
    DumpResult(ret);
    }
    void ExecuteButtonInvoke(string automationID)
    {
    //Create query condition object, there are two conditions.
    //1. Check AutomationID
    //2. Check Control Type
    Condition conditions = new AndCondition(
    new PropertyCondition(AutomationElement.AutomationIdProperty, automationID),
    new PropertyCondition(AutomationElement.ControlTypeProperty,
    ControlType.Button));

    AutomationElement btn = numberzWindow.FindAll(TreeScope.Descendants, conditions)[0];
    //Obtain the InvokePattern interface
    InvokePattern invokeptn = (InvokePattern)btn.GetCurrentPattern(InvokePattern.Pattern);
    //Click button by Invoke interface
    invokeptn.Invoke();
    }

    void DumpResult(string[] ret)
    {
    StringBuilder sb = new StringBuilder();
    int total = -int.Parse(ret[0])*2;
    for (int i = 0; i < ret.Length; i++)
    {
    sb.Append(ret[i]);
    sb.Append("\t");
    total += int.Parse(ret[i]);

    }

    sb.Append(total);

    Console.WriteLine(sb.ToString());
    AppendToLog(sb.ToString());
    }

    void AppendToLog(string line)
    {
    FileStream fs = new FileStream(@"C:\1.txt", System.IO.FileMode.OpenOrCreate);
    StreamWriter sr = new StreamWriter(fs);
    sr.BaseStream.Seek(0, SeekOrigin.End);
    sr.WriteLine(line);
    sr.Close();
    fs.Close();
    }

    string[] GetResult()
    {
    string[] ret = new string[6];
    ret[0] = GetTextResult(textTotal);
    ret[1] = GetTextResult(text1);
    ret[2] = GetTextResult(text2);
    ret[3] = GetTextResult(text3);
    ret[4] = GetTextResult(text4);
    ret[5] = GetTextResult(text5);

    return ret;
    }
    string GetTextResult(string automationID)
    {
    Condition conditions = new AndCondition(
    new PropertyCondition(AutomationElement.AutomationIdProperty, automationID),
    new PropertyCondition(AutomationElement.ControlTypeProperty,
    ControlType.Text));
    AutomationElement btn = numberzWindow.FindAll(TreeScope.Descendants, conditions)[0];
    //Read name property of Text control. The name property is the output.
    return btn.Current.Name;
    }
    }
    }

  5. My analysis program shows the following distribution:

    Length=1, Expected count = 10, actualCount=10, Expected each=3387
    // Detail:
    // 0123:400 means number squence “0123” appears 400 times
    4:3338 6:3288 2:3239 5:3234 9:3350 0:3252 7:3271 8:3237 3:4394 1:3266

    From the result, number “3” is very special. It is more frequent to appear than other numbers. It means there is bug which makes the number not “random” enough.

    Please note, a single check for Length=1 is not enough even if every char gets the same distribution. My automation also checks when Length>1. The following is the result when Length=2. However, since single char distribution is incorrect, other checking is meaningless. (The data analysis code is trivial so I do not paste it here).

    Length=2, Expected count = 100, actualCount=100, Expected each=338
    // Detail:
    // 0123:400 means number squence “0123” appears 400 times
    46:331 64:347 42:317 26:281 65:294 59:287 94:357 44:328 45:321 56:315 60:308 05:321 66:335 67:293 74:291 47:343 78:331 88:350 89:316 97:343 73:401 30:430 07:304 71:305 13:404 32:409 21:297 25:339 52:295 23:426 03:449 38:384 84:305 51:316 18:323 86:346 09:316 35:403 20:344 08:329 41:332 15:292 55:304 53:452 27:331 77:320 76:311 62:323 24:318 49:333 99:330 98:290 85:300 10:287 95:309 39:457 90:319 02:300 75:351 12:336 06:325 28:304 81:316 14:332 40:322 34:440 43:416 80:289 69:356 96:314 63:424 31:465 11:324 37:403 72:315 33:590 04:329 36:413 87:310 91:311 16:317 61:307 48:295 79:315 82:303 19:335 01:293 17:316 22:294 58:330 57:308 83:402 29:305 68:301 50:337 70:331 93:430 00:285 54:290 92:347

  6. oh, it should be, you cannot close it unless you first change the window location when it first starts, or after every roll action. (right click in the title bar excluded). Can I use IDAPro or Visual Studio to read the assembly code?

  7. The code shows:

    01221116 b90a000000 mov ecx,0Ah
    0122111b f7f9 idiv eax,ecx
    0122111d 895435cc mov dword ptr [ebp+esi-34h],edx
    01221121 ffd3 call ebx
    01221123 99 cdq
    01221124 b91e000000 mov ecx,1Eh
    01221129 f7f9 idiv eax,ecx
    0122112b 85d2 test edx,edx
    0122112d 7508 jne numberz+0x1137 (01221137)
    0122112f c74435cc03000000 mov dword ptr [ebp+esi-34h],3
    01221137 8b4435cc mov eax,dword ptr [ebp+esi-34h]

    Relative C++ code is:

    if(random()%30==0)
    {
    return 3;
    }
    else
    {
    return random()%10;
    }

    It shows the possibility of number 3 should be:
    1/30+1/10=4/30 (about 13.3%)

    From the test code, the number 3 appears 4394 times of 33870 run, which is:
    4394/33870 = 12.97%, which almost matches 13.3%.

    Alan, Do you use 13.3% for number 3?

  8. Again:

    012211a7 ffd3 call ebx
    012211a9 99 cdq
    012211aa b932000000 mov ecx,32h
    012211af f7f9 idiv eax,ecx
    012211b1 85d2 test edx,edx
    012211b3 7501 jne numberz+0x11b6 (012211b6)
    012211b5 47 inc edi

    If(rand() % 0x32h ==0)
    {
    total++;
    }

    return total;

    This is why there is a 2% chance to offset the total by 1.

  9. Test result: The product cannot be shipped.

    1. “The summing function is correct”
    Is not. After some correct calculations, the incorrect occurs. The incorrect calculation is always one bigger.
    The incorrect calculation came after: 65, 25, 44, 46, 8, 70, 153, 11, 13, 25 correct rolls (in 10 attempts) so I’d say – very often.
    Not found any patterns here.

    2. “The numbers are random”
    Are not. For 1000 rolls in 3 attempts I get:
    0 – 84, 93, 110 = 287
    1 – 118, 84, 89 = 291,
    2 – 106, 83, 81 = 270,
    3 – 125, 153, 122 = 400,
    4 – 71, 100, 111 = 282,
    5 – 95, 89, 113 = 273,
    6 – 123, 88, 85 = 296,
    7 – 112, 85, 93 = 290,
    8 – 86, 115, 88 = 289,
    9 – 80, 110, 108 = 298,

    Nearly all numbers has similar results. Except number “3”.

    Other issues:
    1. Strange behaviour of close button (“x”). Sometimes it works, sometimes not.
    2. Memory consumption rises. In my stress, test memory started at “1 972 K” and finished with “2 224 K”. I did around 60 000
    rolls so for such amount of attempts it is not big consumption but can leads to memory leaks.

    I used AutoIt script for automation. Here is the code for the calculation test. If someone wants the rest I can paste it here:

    Func Calculations()
    WinActivate(‘Numberz’);
    For $i = 0 to 500 Step 1
    ControlClick (‘Numberz’, “”, ‘[CLASS:Button; INSTANCE:1; Text:Roll!]’);
    Sleep(200);
    $Total = ControlGetText(‘Numberz’, ”, ‘[CLASS:Static; INSTANCE:7]’);
    $First = ControlGetText(‘Numberz’, ”, ‘[CLASS:Static; INSTANCE:2]’);
    $Second = ControlGetText(‘Numberz’, ”, ‘[CLASS:Static; INSTANCE:1]’);
    $Third = ControlGetText(‘Numberz’, ”, ‘[CLASS:Static; INSTANCE:3]’);
    $Fourth = ControlGetText(‘Numberz’, ”, ‘[CLASS:Static; INSTANCE:4]’);
    $Fifth = ControlGetText(‘Numberz’, ”, ‘[CLASS:Static; INSTANCE:5]’);
    $calculatedTotal = $First + $Second + $Third + $Fourth + $Fifth;
    ;check calculations
    If $Total $calculatedTotal Then
    MsgBox(0, ‘Wrong calculations!!!’, ‘Wrong calculations in ‘ & $i & ‘ steps’);
    Exit 0;
    EndIf
    Next
    MsgBox(0, ‘Done’, ‘Done’);
    Exit 0;
    EndFunc

  10. Hi Alan,

    Before we start any testing can we gather more information about this product, it’s context and customer ?

    1. Who is the customer ?
    2. Where this application is going to be used ?
    3. What are the consequences of shipping defective product ?
    4. What time is given to test this product ?
    5. Is this standalone product, or maybe just an part of something bigger ?

    regards,
    Alek

    1. 1&2. The customer is a used car dealer in Buffalo, NY, who uses the set of five random numbers to determine how many cars his sale team will try to sell in a given week. If they meet or exceed the total, they all get an extra bonus.

      3. If the app doesn’t work correctly, the salesmen will be confused, and they many not get the extra bonus.

      4. As I said, you have until 11:59pm on Wednesday (Redmond, WA, USA time).

      5. Both – it can be used standalone, or as part of the Fast Eddies Used Cars CRM suite.

      1. Thanks,

        If possible can you please answer additional questions :

        6. On what OS and hardware this app is going to be used
        7. How long this app is intended to run without restart.

          1. Thanks for all the answers.

            1. “the app generates 5 random numbers between 0 & 9 (inclusive)”

            Yes, there are numbers between 0 & 9 inclusive.
            As for random numbers I have generated 20k sets. I have verified this on 2 different machines (different OS and Box). I had 2 approaches a) 20k rolls in a row without restart app b) 20k with restart after every 1k

            Total counts per number can be viewed here : http://i.imgur.com/w8vJ1.png

            Normal distribution can be viewed here: http://i.imgur.com/BuibW.png

            It seems that number ‘3’ appears more often than other numbers.
            In addition with approach b) (every 1000 rolls restart the app) looks like the number ‘3’ appears more often than with approach a)

            2.”the summing function is correct.”

            For 20k rolls I’ve spotted 400 wrong sum calculations which equals 2% of total. The wrong sum was always larger by 1 comparing to correct sum which looks like a hidden pattern, unfortunately I was not able to figure it out

            3. There is an issue with closing this app. I encounter this on Windows XP and Windows 7

            4. “(…)tell me if it’s ready to ship (according to the stakeholder expectations)

            Hard to say really when keeping in mind that shipping decision is always business decision. All in all we know that this app doesn’t meet all stakeholder expectation because numbers are not random and total doesn’t always work correctly. On the other hand there will be around 50 rolls per year per car dealer where randomness deviation may be not easily spotted. In addition statistically there may be one bad sum calculation a year (2% * 50). Is this acceptable ? I don’t know but for sure doesn’t meet given requirements.

  11. Application cannot be shipped.

    The X button problem is more complex than described in previous comments.

    Summary:
    Aplication does not close when focus is set on it.

    Steps to reproduce:
    1) Open Numberz
    2) Make sure that focus is set on Numberz and move it(moving is not necessary to reproduce this, however it clearly shows that this is not just startup problem)
    3) Focus any other app (i.e. IE)
    4) Set focus to Numberz.
    5) Try to close application with X.

    EXPECTED OUTCOME:
    App is closed.

    ACTUAL OUTCOME
    App does not close. Even when app is moved on the screen and focus is lost of it (and then it is focused again) application cannot be closed. This is not aplication startup problem, but changing/loosing focus problem. After every focus on Numberz – move of aplication is required to close it.
    You can move it, then loose focus, go back to it, and it still won’t close.

  12. Don’t know if this is a bug or a feature but:

    a) Holding Enter causes that “Roll” is pressed and numbers are generated in-loop.

    b) Holding space causes that no number is generated, releasing space causes that numbers are generated only once.

    So for a) and b) we have 2 different actions – this however needs consultation with developer as i’m not sure it it’s a bug or feature.

    Third problem is that application has serious problems with opening it for several times.

    Steps to reproduce:
    1) Open Numberz
    2) Press Alt+tab and focus on catalogue window from which numberz were opened
    3) Press Enter to open Numberz again
    4) Repeat several times rapidly.

    EXPECTED OUTCOME:
    Numberz are opened every time Enter is pressed.

    ACTUAL OUTCOME:
    Windows has tendency not to open Numberz in this TC. It is nearly 75% reproducible after 3rd iteration and 100% reproducible when this test is repeated several times. No memory issues were noticed when reproducing this bug.

    BTW In both my posts I am using Windows XP Service Pack 3.

  13. Two things I noticed:

    The five digits that a roll produces are ‘programmatically’ not in the same order as displayed in the app (digits 4 and 5 are switched). This is not a show stopper though…

    The other thing I noticed is that the same result of the roll (e.g. 3 8 8 6 7) can give a different total. Once I got a 33, the next run a 32 (which is correct). This means that the roll itself does NOT trigger a false calculation.

  14. 1) Total is not always correct and can be off by 1
    Would like to ask the dev how he works out the total – does he add the same numbers he puts in the boxes or is there some rounding error coming into play ?

    2) The numbers are not totally random – 3 is chosen more often than the other numbers.

    App was automated using Ruby – testing manually would not have shown these errors unless the tester wanted to get RSI by pressing the Roll button a lot

  15. Sill trying to get more detail on why the total is not always calculated correctly but here a summary of the testreport:

    The stakeholder has 2 primary objectives:
    1) numbers are random
    2) the summing function is correct

    Testresults for both objectives show that both objectives are not met:

    The numbers are not random. Results after 32000 Rolls (times 5 numbers) show that number 3 shows up more than 13% (while the other numbers rate just below 10%)

    The summoning function is not always correct, in more than 2% of the rolls the total is 1 higher than it should be. This deviation seems to occur at random. Even when the roll results are the same the total might be wrong in one roll while not in the other.

    Besides the primary objectives, the stakeholder will probably want the GUI to behave like ‘standard’ windows GUIs behave. This is not the case, e.g. the X at the right top should close that application, which is does not for this application.

    Advise based on the above findings: Do not ship this application.

Leave a Reply to Alan Page Cancel reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.