Performance Tests July 2015

Skip to end of metadata
Go to start of metadata

Test environment

Test scenarios

  1. HTML file upload through the API
    • Data set of three files
      • textbook-sample-math.html = 5 equations from an O'Reilly book
      • asciimath.html = 10 equations from Chapman site (similar to above)
      • mathmlcloud-tests.html = 110 equations from original sample prepared for MMLC testing
    • Configuration 1: 10 users, each uploading 10 files from the data set (round robin selection)
    • Configuration 2: 5 users, each uploading 50 files from the data set (round robin selection)
    • Configuration 3: 3 users, each uploading 50 files from the data set (round robin selection)
    • New users created for each test
    • 15 seconds pause between each equation
    • 5 minute response timeout
  2. The Node system was restarted before each test run
  3. JMeter script and data files are in GitHub repository mmlc-regression-html

Test results

Response Times (milliseconds) and Throughput (requests/minute)

Scenario Number of samples Average (ms) Median Min Max Throughput
10 users, 10 files 100 68123 24032 1747 150729 6.2/min
5 users, 50 files 250 10730 594 205 157915 11.6/min
3 users, 50 files 150 22432 1165 217 156371 4.8/min

Graphs

Response Time for 10 users, 10 files

Response Time for 5 users, 50 files

Response Time for 3 users, 50 files

Analysis

  • I ran each of the scenarios a couple of times, and results were not consistent for the same configuration, even though the Node system was restarted each time. Our VM configuration could sometime hold up for 100-150 file upload requests before being overloaded and exceeding the 5 minute request timeout, sometimes hold up for 200 or more file requests.
  • Throughput appears to top out at about 11 file uploads/minute, but decreases over time, suggesting that the server either cannot keep up or that there is a memory leak. This value, like response time, was inconsistent across runs.
  • For the first two test runs, after the test run was completed, the MMLC app had issues getting data from the API, and so appeared unresponsive. This seems to suggest an issue with memory menagement that is not allowing the system to return to a stable place.
  • The console log does not show any error output, but some runs began getting timeout errors as the test went. We cannot yet see into the Node server, though, to identify bottlenecks or memory use issues.

Recommendations

  • Scale: Our VM configuration is for a small, single CPU machine. Node.js and Sails.js scalability appears to be best done by running an instance of Node per CPU core. To get better throughput we should explore the options available for scaling Node/Sails applications, and run the tests with the system running on multiple cores.
  • Monitor: To deal with stability issues, we should gather tools that would allow us to identify potential memory leaks or bottlenecks in a Node.js system.
  • Repeatability: These test numbers give one view of the system, but more work needs to be done to get more consistently repeatable results.

Raw Data

Name Size Creator Creation Date Comment  
File Response-Time-Graph-run3.png 32 kB John Brugge Jul 23, 2015 08:10    
File Response-Time-Graph-run2.png 42 kB John Brugge Jul 23, 2015 08:10    
File Response-Time-Graph-run1.png 27 kB John Brugge Jul 23, 2015 08:10    
File request-response-run1.csv 18 kB John Brugge Jul 23, 2015 08:10    
File Aggregate-statistics-run3.csv 0.3 kB John Brugge Jul 23, 2015 08:10    
File Aggregate-statistics-run2.csv 0.3 kB John Brugge Jul 23, 2015 08:10    
File request-response-run2.csv 25 kB John Brugge Jul 23, 2015 08:10    
File request-response-run3.csv 15 kB John Brugge Jul 23, 2015 08:10    
File Aggregate-statistics-run1.csv 0.4 kB John Brugge Jul 23, 2015 08:09    
Enter labels to add to this page:
Please wait 
Looking for a label? Just start typing.
  1. Aug 07, 2015

    John Brugge says:

    I saw this thread on the Mathjax-node site that describes another person's exper...

    I saw this thread on the Mathjax-node site that describes another person's experience with benchmarking.

    https://github.com/mathjax/MathJax-node/issues/107

    One point from a Mathjax contributor:

    Note, however, that MathJax takes time to process the equations, and (depending on the complexity) probably won't handle more than a couple of hundred equations per second. You certainly will not be able to process 10K per second. The MathJax API does queue the requests as they come in, so if you try to do 10K at a time, it will need to store those requests, and that will increase its memory usage.

    And from the person running the test:

    AFAIK, MathML to SVG is a CPU intensive job, so the concurrency of Mathjax-node is limited by CPU but not by RAM, etc. So the concurrency may not be very high.

Add Comment